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ABSTRACT 


quantitative analysis in many areas of science eee is_ ‘emphasized 
in this. report of the -hearing_ on_science evaluation methods_used_in 
Great Britain. The testimony given_ by Professor Benjamin Martin of 
the: Science. Policy. Research_Unit at the. University. of Sussex in . 


England explains how. quantitative. information-is used by the British 


in develeping. policies. for science. Appendices. consist_of-the 
selected papers by Professor Martin and his colleague Professor John 
Irvine and also contain sever critiques of Martin and Irvine's 


methodology. (ML) 
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BRITISH SCIENCE EVALUATION METHODS 


WEDNESDAY, OCTOBER 30, 1985 


a House: OF REPRESENTATIVES, 
CoMMITTEE ON-SCIENCE AND. TECHNOLOGY, - 


Task Force on Science Poucy, ~ - 


by way of introduction that in many areas of science policy we 


would like to see an -increase in- the use- of statistical data and 
quantitative anal sais Such-statistics have in the past been limited 
to information about findings and manpower—or funds and man- 
power, and the analysis of this information has been at quite an 


tinue-to play 
that 


a 
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tell us about the important work that Professor Martin has agreed 


toa Wee ee ts Ae ee ee. ee See 5 one = wee wh = 
We recognize thatthe methods developed in England -have not 
yet. pean perfected. We_are also aware that in some cases the re- 


. There is more than a little irony in the fact that while science 


itself has been affected-so immeasurably from the application of 


th pe to emonstrate. = eo eee See, 
small team at the Science Policy Research Unit, 


oping Systematic. methods for evaluating the research performance 
0 Siete oe eee 


tronomy... - eee ee, Reem See, See hs Gee es, (oe REAR ERE eens 
_As I shall shortly, the overall approach adopted in-such 


searchers in different countries who are asked to rank these facili- 
ties in terms of their_relative contributions-to-scientific knowledge 
and to explain the factors determining differences_in performance. 


sioned—for example, bythe British Advisory Board for-the- Re- 
search Councils and various learned societies—and covers such 
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fields as protein crystal ysta liography and ocean Pent currents as weil as-Brit- 


ain’s_overall scientific. performance relative to other countries. Re 
cently- published_results on. Britain’s declining scientific perform- 


ance attracted: significant. public interest.in- the UK -earlier-this 


aaa notably during debates in the House of Lords and House of 


evaluations of past performance, we have begun to explore methods 


later. SG. +e 2 2 Bae NS Sh Beas Seal i ee BL 
re importantly, the question of how one selects long-term re- 


The.aim is to produce not just totals foreach country, but also.a 


breakdown by 9 fields and 40 subfields. This is a very topical sub 
ject at present since senior policymakers are increasingly interest- 
ed_in knowing how their spending compares with that -in- other 
countries, and the international statistics available at present are 
somewhat limited _technically, and in their level of disaggregation. 
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view. To illustrate how-eval ms of research o1 rt 
arried out, I shall-describe a study which compared the scientific 
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these scientificcontribution. == = = = 
Second, the approach here is institutionally-focused—that is, the 
unit of-analysis is the research laboratory, facility, or_group—be- 


ting a slightly different facet of research -per- 
looks- at -the- number of scientific 


finding is reported in a journal article—then the numbers of such 


apers. A typical. paper-will cite -10-or 20 previous papers which 
have had some impact on the: work it-reports. Thus, the important 
ighly cited. For example; the most hi cited 
paper in experimental. high-energy physics during the 1960’s was 
also the only one which subsequently led to a Nobel Prize. -- - -- 


Lest, data on the distribution of highly cited papers reveal which 


additions to knowledge. aia 


duced. From table 1—the tables are in appendix 1—you. can nee 
that in 1959 the CERN Proton Synchrotron, or PS, in the joint Eu- 


nF 
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fornia, as the world’s highest energy accelerator. 


TABLE 1 


The world's main proton accelerators (25 GeV) 


Accelerator 


CERN PS (W.Europe). . 
Brookhaven AGS (U.S.) 
ITEP Moscow (U.S.S.R.) 
Argonne ZGS (U.S.) 
Rutherford Nimrod (U. K. ) 
Serpukhov (U.S.S R.) 
CERN ISR (W.Europe) 
Fermilab (U. S. ) 


CERN SPS (WwW. Europe) 


CERN. pp (W:Europe) 


1967 when- ‘the Serpukhov. accelaratar wes completed, ‘but nel 
to the-United States in 1972 -when-_the- Fermilab_accelerator began 


operating, 4 years ahead of the similar SPS accelerator at CERN. 
-Last, the CERN proton-antiproton, or pp collider started oper- 
ation_in 1981; and this remains-the world’s most powerful facility 
to. date, although the-larger Fermilab Tevatron Collider is very 
shortly to begin experimental work. 
-In. analy the outputs-from. the docdlerators,. awe- divided - the 


lyzing 
period from 1961 to 1982 into 4-year blocks. Thus, table 2, covering 


i2,,: 
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1961 to 1964, shows that the Brookhaven AGS produced 4:5 percent 


of the world total of experimental high-energy physics papers in 
the first 2 years and-11_-percent-for 1963-64, somewhat behind its 


main rival, the CERN PS with 29.5 percent, and the older Berkeley 
Bevatron with 23 percent. 


Highly cited papers : 


number cited n times! 


Bevatron 


1. Over the period 1961-80, an experimental-paper-required approximately. 
- 40 or more Citations ‘in any one year to be_included_in-the top-1£-mcs 
highly cited_papers for the two decades, -and 19 or more to:be include 
inthe top 5%. In-this. respect, n315. corresponds_to the_top 7:85, © = 
ne30 to the top 1.9%, nb50 to the top 0.6%, and n4100 to the top 0: 14% 
most highly cited papers. 


2. ALL. figures fn this and subsequent tables have been rounded to the nearest 


AGS and the Bevatron, the respective world-citation shares being 
14.5 percent, 23 percent,-and 35 percent in 1964... -... 
On average,-each AGS paper was cited 8 times in 1964, a typical 
figure for a new. accelerator, while CERN PS publications earned 
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identification of two types of - reutstne: the omega -minus, aad 
charge-parity violation—were all -made on the Ue S. accelerator. 


19.5 percent compared “with 25.5 percent in 1968—but those papers 
were still more highly cited, averaging 4.6 citations per-_paper-in 
1968 compared with 3.1 for CERN publications, with the result that 
the total impact of the two machines seems to have been very simi- 
lar, at least in terms of world citation share—the figures were 26 
and 26:5 percent in 1968: 


Experimental high-energy physics: 1965-1963 


Average-- Highly cited capers: | 
_ [citations a Se, ep ee 
ber paper nomber cited n times 


| 
| 
{ 


_ Rutherford 
‘ Afwrod - -- > 


gas [4500 |5080 i 3.8 
100d 


- For the. period 1969-72, table 4-reveals that while-the- PS pro- 
duced. a- steady 25 percent-of world experimental publications, the 
AGS’s share dropped to 14.5-percent. in 1972, following-difficulties 


associated with a major. technical upgrade of ‘the accelerator. How- 
ever, the most highly cited papers at this time camé from neither 
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of these accelerators but from newer-machines. The AGS and PS 
each managed only_one paper cited 30-or more times, while Ser- 
pukhov -in the Soviet Union, the CERN ISR, and the Stanford 
Linear Accelerator, SLAC, each yielded four. 

Experimental high-energy ohysics: 1969-1972 


Highly cited papers: | 
numer cited n times 


years |- 
1970 |1972 ]1970 11972 | 1970 |1972 Ina 


Bevatron 


_ The next 4 years, 1973 to 1976, were among the most tumultuous 


peat 
On 
“ i} 
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Highly cited pacers : 


number cited n tines 


9.0%| 6.0%| 7.0%] 4.52| 2.3] 2.7, 


was. turned down 


result, Stanford had to resort to building the machine from its op- 
erating budget. - fi 2 - - : ase 


16. 
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11 
period, partly as.a result of funding problems, and partly because 


of the growing impact. of its European rival, the-new CERN SPS 


which, taking advantage of beams and detectors signifi icantly better 

than those_at-Fermilab, achieved a particularly high rate of cita- 

tions per paper, 12.7 in 1978 compared to 7.3 for Fermilab. 
TABLE 6 

_Experimental-high-eneray physics, 1977-1980 


Highly cited papers: 


3. OF fof ‘| Average _ 
papers. citations | citations 


number cited a times’ 


; ae ee 
ealeelee 2.7] 16/010] 4 


Serpukhov en ee ee ee es ee = 
a: 412.05 114.0 | 4.0%] 5.08] 1.2) 1:2 | 0 : 0 


CRW ISR / j | _t_ ft __ id: | = 
ay ‘ 
| 


Rest of are a a ee | 
world- 23.52 |24:05/13.05/15.0% 2.01 1.9 | 19 


To: 


As in-previous periods, though; the crucial discoveries were made 


in the United-States-rather than at CERN, the two papers cited 


over 100 times during the 4 years reporting the discoveries of the 
upsilon at Fermilab and of parity violationat SLAC. .  ._. . 
- Finally, what has happened since 1980? Comparison of table 7 


with earlier tables shows that Fermilab’s world share of citations 


had fallen from a peak of 32 percent in 1978 to 16 percent in 1982, 
while the figure for SLAC had likewise dropped by nearly half be- 
tween_1976, when it was 17 percent, and 1982, when-it was 9 per- 
cent. Overall,-the- U.S. share of citations has..decreased sharp 


from 59 percent of the world total in 1978 to 34 percent in 1989, 


tim. 
ae 
fof, 


physics, post-1980 


pacers 
published 


Cornell CESR 
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13 
_-Conversely, if the figures for-the various accelerators at CERN 


and the German: laboratory, DESY, are combined, they suggest 
that the early 1980’s heralded a European. renaissance in high- 


produced 80 percent of the papers cited.15 or more times-in a-year, 
while the U.S. share was-just 20 percent, a-complete reversal of the 
situation inthe period -1973-to 1976 when the two-European labora- 


percent. - 


Overall rankings. 
(sample size = 169) 


6.9(49-1) 
6:1(+0.2) 


Cs IC 
Ea EX 


B.5( 
emit au09-|eauni 
rain | esa 
1, T0=top. The assessments are based on-the relative outputs from. 


the-accelerators over their entire operational careers up-to the 


More ‘precise 
measurements 


time of the interviews with high-energy physicists :in late 
1981/early 1982. : 


ments of different groups of researchers. While there was a slight 
self-ranking effect, a tendency to rate one’s own work more highly 
than others do, this was not very significant except in the case of 


ig” 
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Se Serpukhov. The fepults show ‘that the Brookhaven AGS « was ranked 
ahead of its-European rival, the -CERN-PS,-in terms of discov- 
eries; in line with the dé ta on highly cited papers. ‘tae 
ppengillaely, the-Fermilab accelerator,-which generated 1 ae PAOy. more 
highly: cited papers than the equivalent-energy CERN CERN- SPS, - was 
sanked significantly ahead of it. In contrast, for precise measure- 
ment experiments, the PS was-ranked ahead of the PAGS, consistent 


with the data on total citations, and the SPS was likewise placed 
ahead_of- Fermilab. 
ee interpretation of the picture 2 ieee by-the. various-in- 


d sings. It is ‘hee nee 
vith the tee comments: of. f highen gr rays ‘on our results, which 
= 


TS can - 7 out. evaluations: of 
past research perfortaance within individual basic science special- 


ties. In the case of the above study, among the conclusions that we 
arrived at were the following: _ 


cial discoveries in sxporinental ‘high energy physics t etween 1961 


and_-1982-were-made at U.S. laboratories, even though similar 

energy accelerators were generally availabl available-in Western. Europe. 

The situation has, however, c dramatically since 1982. __ 
-Second, as 1 gards- experiments producing precise measurements 


and, results \ wit! gett — the Ben record. of the machines 


rE beven't tia time to give 5 details ee on: “the. here the record of 
oe ie US.. national laboratories and Stanford in: particular 
seems to have been { HN theis meer , to-have- been. oo 


better than that of C NV, their-main European ia a ge 


reine hk peer review proses fandat aad. ‘ranoparen, ; 
es the pos the- perform: 
undertaken. aad 


were most neeied a or, aie natively, where commitments soul be 
reduced in order to free vunds to to support new areas and new 


people. =p 


Last, a8 we describe. elsewhere, the method makes p asible a sys- 
tematic appraisal ofthe prospects. fora-major. new basic research 


facility compared with rival facilities woridwide. In the case of our 


0 
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is completed first ut CERN. - .- -- Lor =: 

—In-conclusion, as industrialized nations around the- world have 
moved into an era of approximately level_science budgets, so. the 
task of deciding which of the competing claims of researchers. for 
new projects should be given priority-has become more difficult. If 
policymak cers were to carry- out. quantitative science policy studies 


similar to that outlined here, these would in our view provide them 


with information of direct relevance to this crucial task of estab- 
lishing. scientific priorities. 
k you. . = Bee min ee ee ee gi ere oe ee ee 
[The prepared scatement of Professor Martin follows:] 
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John Irvine and Ben R. Martin" 
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To be presented to the Tast Force on Science Policy of the Committee on Science 
and Technology; U.S. House of Representatives, 30 October 1985. 
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No_order of seniority implied (rotating first authorship): The: authors..:-.. 
are Fellows of the Science OA pe fecoigpesinpe et pe bigger cer _of Sussex, 
where they work on a range of issues connected with government_policies for-_ 
basic and ppl ied research. They gratefully acknowledge the financial viol id 
of the Leverhulme Trust in carrying out their current programme of research. 
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QUANTITATIVE SCIENCE pC PGLICY RESEARCH 


Introduction: general overview 


Since 1978, 11. team. at the Science. Policy Research Unit (SPRU) has- carried 
out a range of quantitative Science-policy research studies. Much-of the effort - 


has been concentrated on developing systematic methods for evaluating the research 


performance of laboratories; facilities and groups ._ Initially this work 


focussed on basic science, {n particular on the large central facilities used to 
carry out-research-in-'big sciences’ such-as high-energy physics (see refs 7, 17; 
18,-and_19 listed in Appendix II), optical astronomy (ref. 12) and radio astronomy 
<ref. 13). 


As we shall demonstrate below, the overall approach. adopted in i institutional 


evaluations involves the combined use of (1) a range of science output. indicators 
(based.on publication. and citation-analysis) to assess the ‘productivity’ and 
*{mpact*® of a-research facility-relative to-similar facilities elsewhere, and- 


(2) the results of structured interviews with a-large sample of researchers in 


different countries who are asked to rank these f facilities. in. terms of. their 
relative contributions to scientific knowledge, and to explain the factors 
aeeereionia differences “ performance. 


First, besides waitin the. sutfats ¢ from research insti tuttons, we. have started 


to develop a programme of work on the assessment. of national scientific. - Performance 


in different fields and subfields of science, especial ly those. with strategic. 


technological importance for the future. This work has mainly been iaumisiioned 


(by-the British Advisory Board for the Research Councils and various Learned 


Societies) and covers such fields-as protein crystallography and ocean currents. 
(see refs ..21 and.32). as well as. Britain's overall scientific -performance-relative 


to other countries; Recently published results on Britain's declining. scientific 


perforn-ice (see refs 27 and 39) attracted significant public interest {n the UK 


earlier this year, | notably during debates in the House of Lords and 
House of Conmons. 


Second, a number of studtes comm ss {oried- by matty. foreign governmental agencies 


took us into the evaluation of a applied R&D_activities. These include projects - 


carried out for a Norwegian Royal Comnissfon.on. Industrial Research (we assessed 


the performance of research institutes and support mechanisms for electronics ard 
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mechanical. engineering - see refs 42. and 43) and for the : European Commission 


refs. 45 and a); 2 as well as preparatory work for @ current review of British 


support for engineering research by the Science and Engineering Research Council] 
(ref.-50). In evaluating applied research, the approsch normally involves 
applying.a-somewhat different. set of indicators based upon industrial and tech- 
nological. impact. although publication and citation statistics were used-in a 


recent stady comparing international performance in the field of integrated 


optics (ref. 33}. 


Third, in .ine with the needs of policy-makers to move beyond evaluations of past 
performance, we have begun to explore-methods for systematically appraising the 
future prospects Of major new. facilities suchas accelerators (see ref. 19), 
work which 1s discussed briefly. below. . More- importantly, the question of how 


one selects long-term priorities in the most effective manncr was taken further 


in a 1984 study for the British Advisory Council on Applied Research and . 
Development which has become. Anterested An the question of. strategic. noni toring 


Picking the Withers (ret: 23), involved. a survey. of state-of-the-art methods used 
by governments , research-funding agencies. and. high-technology firms to formulate 


their priorities for longer-term strategic research. 


aces over the last year we have initiated a ely ew of work on the 4 "Anputs, 


those on national scientific. performance. in particular felds) has Sect ever 


more apparent over time. We are currently engaged in a major study to compare 


government funding of academic and academically related research across six 


countries (the United States, Japan, West Germany, France, the United Kingdom 

and the Netherlands). The ata{s to produce not just totals for each country, 

but 31S0 @ breakdown-by 9 fields and 40 subfields. This is a very-topical subject 
at. present since senfor policy-makers. are increasingly interested. in. knowing how 


their spending compares with that in other countries, atid the available inter- 


national statistics are somewhat limited. technically. and in their level of dis- 


aggregation. This study is largely being financed by the British Advisory Board 
for the Research Councils who wish to know-whether Britain's apparent decline in 
scientific performance {1s related-to relative Tevels of funding across countries. 
It will be completed early in 1985. 
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In order to illustrate the type of approach used in research evaluation, we 
thought it-would be of greatest utility to the Task-Force to focus in detail 
upon one of our recent studies of ‘big-science’ facilities-since the question 


of evaluating institutional. productivity, - while often politically contentious, 


is in our view a central question facing those concerned with science - policy. 


(Details of our other work can be found in the papers listed in Appendix Il. .) 


First, however, we should review briefly some of the msin reasons behind the 
need for better techniques for evaluating research performance. 


The need for research evaluation 
In Western countries, the main decision-making mechanism in basic science con- 
tinues to be peer-review - that is, relying on the opinions of scientists in 


the field concerned.- However,-peer-review faces severa] problems of growing 
importance, especially.in relation to ‘big science’. - One stems-from the-treid- 


away from the rapids increasing national science. budgets seen in. the 1950s anid 


‘60s towards approximately level funding: Under the latter conditions, 


decisions to reduce existing financial commitments often first have. to be made 
in order-to free funds to support promising new research areas and young scien- 
tists. --This-is not a task for which peer-review is. always very effective 
(ref..16). Second, as the costs Of certain-research_have escalated and resources 
have become concentrated in fewer laboratories, sO it. has become harder to locate 


‘disinterested’ peers able to provide expert judgements on proposed new projects 


but whose own circumstances will be unaffected by the subsequent funding decision: 
Third, because specialties generously funded in the past are generally well 
represented on decision-making bodies, peer-review has exhibited a tendency to 
reproduce past-priorities and-to be less successful at identifying newly emerging 
(especially interdiscipl inary) research areas. 


Because of these prob lens; there. isi we would argue; a aaed for more ayitaeati 


output data to complement but-not_replace peer-review. To illustrate how evalu- 
ations -of- research. output might -be carried out, we describe a study which compared 
the. scientific performance of- - high-energy physics.accelerators around the world, 


including those operated by the main US laboratories (see refs 17-19, 28 and 30). 


The method of converging partial indicators 
How can one evaluate research performance in basic science? There are four main 
elements to our methodology. First, it is based on an input-output approach - it 
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isvotves Aéentt fying: and olla the far ae (funds, researchers, ete.) 


possible since the primary output is cortriwutioal to scientific knowledge, and 


we therefore focus here on the evaluation of scientific contributions. 


Second, the approach 1s institutionally focused 1 (he: the unit of sf amaiysis is the 
research laboratory, facility, or group) because major. investment. decisions_in .- 


basic Science normally centre on institations, and it 4s here that rigidities with 


peer-review most often arise. Third; because no absolute quantification. of 
research performance is possible, the approach is comparative, with the proviso 
that one can only beneny: baci "like' with ‘like’. 


Fourth, . the approach involves the. comb ined. use of F several indicators;. each. 


reflecting & slightly different facet of research performance. For example, 


publication totals (that is, the number of scientific papers published in inter- 


national learned journals) give some indication of the scientific production of a 
research group, while numbers of. papers per researcher-or. per dollar reveal- some- 
thing about its productivity. The average number of citations per paper (that is, 
the average numbrr of times each paper is referred.to.or.‘cited'. by other. 


scientists in. subsequent papers) gives an indication of the impact those publi- 


cations have on the scientific community, while peer-rankings (where scientists 


are asked to rank the relative contributions of different institutions) provide 
evidence on the perceived significance of. the results. -Last, data on-the distri- 
bution of highly cited papers reveal which groups -have been responsible for the 
few. key ‘discoveries’ in a-specialty, while citation. totals reflect the. large 


number of - Ancremental additions t to knowledge. It. should be. stressed that a1] 


these indicators are imperfect or ‘partial’ - they reflect partly the relative 


magnitude of contributions to sctentific _knowtedge, and aghad a variety of "social, 


international literature siibject to comparable- referecing. procedures, the L 
indicators have been found.in previousstudies to yield. broadly convergent results 


(see, for example, . ref. 12); Such results we regard as being reasonably reliable 


- certainly more reliable than those based on a single indicator like peer-review. 


Our sem. focussed on the world's main proton secelerators; a1 though cata a dats 


relating to electron accelerators were also produced... From Table 1, one can . 


see that in 1959 the CERN Proton Synchrotron (PS) took over from the East European 
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Dubiia accelerator, and before that the Berkeley Bevatron-in-California, as the 
world's highest-energy accelerator. However, the very similar-and slightly 


higher-energy Brookhaven Alternating Gradient Synchrotron. (AGS) was completed 


a few months later in 1960. The lead passed to the Soviet Union_in 1967 when 


stmt lar SPS accelerator at CERN.. -Last, the CERN proton-antiproton (pa) collider 


started ee in_1981, and this remains pies ewer la" S most t powerful. facility: 


experimental work. 


In. anslyzing. the. outputs from the accelerators, we divided the period 1961-82 
into. four-year ‘blocks‘.- Thus, Table 2-covering 1961 to-1964 shows that the 
Brookhaven AGS produced 4.5%. Of the world total of experimental high-energy 


physics papers over the first two. years and.11% for_1963-64, some way behind 


its main rival, the CERN PS (29.5%), and the older Berkeley Bevatron (238). 


However, the CERN papers reported data from relatively simple experiments, 50 


thei> overall impact (as reflected in the number of times they were cited by 
Other scfentists) was Jess than-that-of the -AGS.and-the Bevatron, the respective 
world citation-shares. being 14.5%, 23% and 35%-in 1964. On average, each AGS 


paper was cited 8.0 times in 1964, a typical figure for a new accelerator, whitle 


CERN PS publications earned only 1.9 citations per paper. Moreover, in terms. 


of highly cited papers, the Bevatron and AGS both seem to have been responsible _ 


for more discovertes than the-PS. indeed, the three major discoveries that could 
have been Wade on either the AGS or the PS-- the identification of two types of 
neutrinos, the omega minus, and charge-parity violation - were al] made on the 


US accelerator. 


Over the next four years: Tabie 3 shows that. es gap between she MS 2 and PS 


narrowed. - The AGS continued to yield fewer papers (19.5% compared with 25.5% 


in 1968), but.those papers were still more highly cited (averaging 4.6 citations 


per paper in.1968 compared with 3.1 for CERN publications), with the resutt that 


the total impact of the two machines seems to have been similar, at-least in 


terms of world citation-share (the figures were 26.0 and 26.5% in 1968). For 


papers cited 15 or more times in a year, the AGS had by then been overtaken. by 


the PS, but the AGS continued to prove much more successful as regards the more 


from the PS: 
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For the period 1969-72, Table 4 reveals that, while the PS produced a steady 
25% of world experimental publications, the AGS‘s share-dropped to 14.5% in 1972 
following difficulties associated-with a-major t-chnical- upgrade of the 
accelerator. However, the most highly cited papers at this time came_from 
neither of these accelerators but from newer machines. . The AGS and PS each 


managed only one paper cited 30 or wore times, while Serpukhov, the CERN ISR 


and the Stanford Linear Accelerator (SLAC) each yielea four. 


The ext four years 1973-76 were among. the most tumultuous ever in- High-energy 
physics, ushering in the era of ‘new physics’. CERN made.a promising start in 
1973 with the ‘neutral currents‘ discovery and. several other major advances, 


but after this most of the important discoveries were made in the United States. 


In terms of papers cited 15 times in a year, the PS and ISR were well behind 
SLAC and the new Fermilab-accelerator near Chicago, while no less than five out 
of-eight crucial discoveries (cited. 100-times in a-year)come from the SPEAR. 
collider at-Stanford. _(It may be of interest to the Task Force to know. that. 
the proposal to. build this highly. novel. and successful accelerator was turned - 


down by the US high-energy physics. community on several occasions; illustrating 
the conservative tendencies implicit in peer-review. As a result, Stanford had 
to resort to the somewhat dubious procedure of building the machine from its 

Operating budget.) -In-addition, it was the Brookhaven AGS rather than the. CERN 
PS which shared. with stanford the honour.of making arguably the. most important 
advance of. the. 1970s - the discovary of the 3/psi particie which paved the way 


for the ‘new physics’: 


In contrast, the following four years were a period of consolidation. According 
to the figures-in Table-6, the Fermilab.accelerator seems to have contributed 
most during this time with about a-quarter of total world-citations and. 40 papers 
cited 15 ormore times. towever, its. performance. did decline markedly. tovards 
the end Of the period, -partly as a result of funding problems, and partty because 


of the growing impact of the new CERN SPS which, taking advantage of beams and 


detectors significantly better than those at Fermilab, achieved a particularly 

high rate of citations per paper (12.7 in 1978 compared with 7.3 for Fermijab). 

As in previous periods, the crucial discovertes -were-made -in-the United States 

rather. than. at CERN, the two papers Cited Over 100 times during the four-years 

reporting the discoveries cf the upsilon at Fermilab and of parity-violation at 
SLAC. 
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Finally, what has happened since 1980? Comparison of Table 7 with earlier tables 
Shows. that-Fermilab's world-share of citations had fallen from a peak of 32% in 


1978.to 16%. tin. 1982, while the-figure for-SLAC had likewise dropped by half 


between 1976 (173) and. 1982 _(9%).- Overall, “he_US share_of citations-has decreased 


sharply from 59% of the world total in 1978 to 34% in 1982. Coriversely, if_the 


figures for the Various accelerators at CERN and at the German. laboratory, DESY, 


are- Sonbined, they Suggest that: the early 1980s heralded a European renaissance 


particles. at. CERN in. 1983. In partici, these wo laboratories produced 80% 


‘of the papers cited 15 or_more. tines in a year, while the-US- share was just 20%, 


& complete reversal of the situation. in the period 1973-76 when-the two European 


laboratortes produced ds Of such papers and US laboratories 68%. 


How_do. all. these bibliometric -Yesults compare with the assessments of nighenanay 


physicists?. 182. researchers friu.1] countries were asked to assess six proton 


accelerators on a 10-point scale.(1G=top) in terms.of (1) ‘discoveries’ and 


(2) experiments involving | more precise "measurements of known particles and pro- 
perties. The results are given in Table 8._ Overall, we found reasonable. consis~- 


teney between the assessments of different groups of _Fesearchers: While there 255 


than 4 do others. -£ this was not very stgnttieant (except-in the case of Serpukhov). 
The results show that the. ‘Srookhaven AGS was ranked.well above its. European rival, 
the CERN PS; in terms of discovertes, in line with the data on highly cited papers. 


Similarly, the Fermilab accelerator, which generated many more highly cited papers 


than the equivatent-energy CERN SPS, was ranked <ignificantly ahead of it. In 


contrast, for ‘precise measurement’ experiments, ihe PS was ranked ahead of the 
AGS, consistent with the data on total? citations, and the SPS was likewise placed 


ahead of Fermilab: 


Although interpretation of the picture yielded by: the. various. tndteators. ts by no 


papers subsequently shown to be ‘mistaken'), there is a certain consistency 


between.the_biblfometric data and the peer-rankings. It js this; together with 


the coments of high-energy. physicists on our results, which leads us to conclude 


that outsiders. can carry out evaluations of. past research performance within 


individual basic-science specialties. In the case Of the above study, among the 


conclusions that we arrived at were the following: First; with one_or. two 


prominent exceptions, nearly ali the crucial discoveries in experimental hi gh- 


energy physics between 196] and 1982 were made at US laboratories; even though 
similar. energy accelerators were -generally-avaiiabie in Western Europe. The 
situation has, however, changed dramatically since 1982. Second, as regards 
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cipariuants productag precise measurements and em with high statistics; t the 
overall record of the machines at CERN taken together has been better than that 
of the accelerators at any laboratory in the United States. Third, in terms of 
overall scientific productivity - that {s, scientific performance adjusted for .- 
the. differing. size of. the various centres (see Tables 8,- 10. -and 4 in reference 1¥) 

- the record of the three US National Laboratories and Stanford in. particular. 


seems to have been significantly better than that of CERN, their main European 


ee ad tor. 


What. : ist the wider eigniticnee of our ancien to itera’ basic science?. First, 


it yields information on scientific performance in.a form accessible not just 1 to 


researchers in the specialty concerned, but also to other scientists, science 
policy-makers, politicians, and the public. It therefore provides a mearts of 
keeping the peer-review process ‘honest’ and transparent. 


Seconds ¢ the method raises the possibility of f tracking t the performance of. any major 


research facility. This can be undertaken relatively quickly. and cheaply ofice 
tne initial data have been obtained. The results could help policy-makers spot a 
significant decline in-the performance of a major research centre (for example, 
because of instrumental obsolescence), and suggest where cash infusions were most 
needed, or alternatively where commitments could be reduced in order to free funds 
to Support new areas and people. 


Last, as we describe elsewhere (see ref. 19), the method makes possible a systen- 
atic appraisal-of the prospects for a major new basic research facility compared 
with rival-facilities worldwide. In-the-case of CERN, this-involved two additional 
stages in the analysis.- First, we identified the. factors - -Structuring success and 
fatiure in past research performance. (The most important factors reported. by 


interviewees as accounting for the differing performance of the CERN SPS. and 


Fermilab machines, for example, are shown in Table 9.) Second, we analyzed the 
various factors to ascertain which-are likely to continue influencing reseerch 
performance in the future,-and to identify any new factors which may emerge over 
coming years. On this-basis,-a-set of..criteria was drawn-up to compare. the. 
prospects for new accelerators of the next decade; including the Stanford Linear 


Collider (SLC) and the proposed Superconducting Super Collider (SSC). Our analysis 


(which was carried out in 1982) led to the conclusion that the Stanford machine 
combined considerable scientific potential with relatively low cost. -The SSC, in 
contrast, -is an-extremel, expensive project, and whether its scientific potential 
will prove proportionately greater. is by.n0.means_certain, particularly ifa 


similar but slightly lower energy collider is completed first at CERN: 


In conclusion, as industrialized nations have moved into an era of approximately 
Tevel Science budgets, so the task-of deciding which of the competing claims of 


researchers for new projects should be-given priority has become more difficult. 
If policy-makers were to carry out. quantitative. Sctence policy studies. similar to 


that outlined here, thess would in our view provide them with information of 
direct relevance to this crucial task of establishing scientific priorities: 
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TABLE -]. 


The world's main proton accelarators (> 


ea eave See t Began 
Accelerator: operating 


Berkeley Bevatron (U.S.) 
JINR Bubna (£.Europe) 
CERN PS (W.£urope) 
Brookhaven AGS (U-S.) 
ITEP Moscow (U.S.S.R.) 
Argonne 26S (U.S.) 
Rutherford Nimrod (U.X.) 
Serpukhov (U.S.S.R.) 
CERN ISR (W.Europe) 
Fermilab (U.S.) 


|__CERN po (W.Eurooe) - 
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Bof - 
papers : : 
published. |to.work of. per paper number cited n timest 
in past 5 3 
two years |four year 


Highly ciced papers: 


Bevatron . 


1962 1964 | 1964 | 1964 |nais |na30 |»a80 [psto0 


Brookhaven 
AGS 


2.8 2}-6| 1 0 


2.8 | | iz} ¢ 1 


Over the period 1961-80, an experimental paper required approximately 

4 or more citations -in any one year to be included in_the top 1% mast 
highly cited papers for the two decades, and-19-or more to-be incluce 

in the top-S%. In-this-respect,n¥I5 corresponds_to the top 7.9%, 

n330 to the top 1.9%, n250 to the top 0.6%, and n3100 to the top 0.14% 
most highly cited papers. 


ALL figures tn this and subsequent tables have been rounded to the nearest 
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TABLE 3 


Experimental high-ene 


Sof leaf --.- Average Highly cited capers: 
papers:  —s_- | citations |ei cations a 
published j|to work of Iper paper rumber citad n times 
in pasts past: 

two years | four years | _ : | : 


--J..y. .t 7 Aaa j 
1968 | 215 | nage 1350 j 


Sevatron 


32:52 25.5% 


Brookhaven 
AGS 


Argonne 26S 


world total ' eas | gas }4so0 | $080 | 
+1008! 100%! 100s! ~ to03i 


56-399 O—B6——2 
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pecie 


Eerivantal high-energy — 1969-1972 


ZS of- Zof _ | Average Highly cite paper: 
papers _—_ [citations | citations ee pee 
published |to work of |per caper number cited n times 
past - 
four years 


1970. | 197% 


Brookhaven Et 


119. 02/14; sz (3 23:08 18. 0% 


World total 950 'iu26- | §270- 5310-| 2:9 
| 100%| 100s! ions, ii 
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Hichiv etted pacers : 


citations 


pe: paper 


| SERN Ps ae ee ee ee 
; - 08 |23. 14.5% 2.7 


Argonne ZS 


Numer cited n cimes 


nN, 


eons as: oy 


era 
+ —a= 


Rutherford: 


tlimrod 
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TABLE 6 


= of % Of: 
papers. - {citatians: 
published 

in-past-- |past —.- 
two" years four_years 


Average 
citations 


Highly ci tea papers: 


numer cited n times” 
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[r=i0 | 
| 
8rookhaven | | i | : 
AGS - | a a ae 
$.$t| $.5t] 3.0%] 3.9% 2.7/ 1.6 | 0 | o | o | 4 
Serpukhov me ae ee | | a 
aes 12.08 (14.0 | 4.08) $03 1.2] 1.2] 0 | 6 | o | oO 
CERN ISR poteyee 2 ae aoe eee l 
meee g | 4.52] 6.55] 7.057.591 $4] a [ar | 2; 0 {jo 
Fermi 146 a Peed ae ae ee ee Cn eee ee ee 
16,94 19-08 (32-05 |21.55 7:3/ 2:8 [eo jie | s | 4 
CERN SPS Boal sah so SS nS eh ee | — ae e | | a 
a 2:5%| 8:52! 4:0] 8:59112:71 s:o0 | 19 7 3; 0 
SLAC : : : 
6 | 1 | 1 
OESY siles. Az | : 
4.02 [6.5 5.515.591 $.7/ a. | 36 E 1a! 
| 
Se Ss aa 
Rest of 2 ee | 7 ee 
world---__|23.5%|24.0%)13.0% 18.03 2.0: 1.9 }19 | § j 0 | 0 
World total. | 1115 | 930-/e190-|s090| 3.5] 3:0 [ies [aa ita ! 2 
= 100% | 100%] 10031 100% 1° | 
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TABLE 7. z 
| Expericantal hi h-ener physics, post-1980 


Sof Sof: .: | Average Hf ‘ey 
papers___ | citations | citations | ¢}5ed 
published | to work of | per paper numer 

past four fi times 


years 


CERN PS. 9.5% 


CERN SPS 


ae 1.02 | 2.0%] - | 208 | : | 9:3 e) 4 
—_ 8.55|10.08| 9.02] 4:01 4:4] 1] 7 | 


ae Fon | 9.00/16.52 19.88] 7.5 8.0 | é| ° 
: : | 


Cornell Cesk | 1.0" | 3.02] 4. | 18.2 a3 2 | 0 
Rest of | : | | = | 
world ee ee 1:3] 01 0] 
worla total | 430 | 390.}si90 | 5070. 2.7 7 2. a! 4s 4 | 
100%} 100%! 190%! 1coz | 
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Mr. Fuqua. Thar you very much, Professor Martin. = -—-- 
. From your statement, you indicated that_a_significant part of 


Britain... __ _- 


-- "fo. what extent to date have they found actual use and applica- 


presentation in fro t of the committee. -. gnats acupad. eeheles swenenes ves lee 
_ Again, I don’t know exactly how much influence our findings 


rely on high-energy physicists to. tell them how successful CERN 


Mr. Fuqua. I notice in your statement you have been doing this 


_.. Professor Martin: Yes. As I said, when we began our work there 


operated by the Norwegian Council for Applied Research and look 
at the work which was carried out at that institute which is sup- 
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posed to provide longer term research of relevance to the needs of 


industry. - _ 


area, and by interviewing the researchers at those institutes and 
also interviewing R&D managers in the two industrial sectors_con- 
cerned, we were able to build-up-a picture as to the utility of the 


~ Now-I suppose the best test.of how. useful that work was is—it’s 
a bit like politicians: do you: get invited back a. second_time?-We 


call saying, could-we come back and within 6 weeks-produce a 


and we were in that case. = _- — ast 
- Mr. Fuqua. Let’s get on another subject to see if you get invited 
back. fLaughter.] SP. es eee ee ee a oe HERS 
On page 3 of your prepared statement, middle of the page, on the 
need for research evaluation, you cite three issues- relating to peer 
review. Peer review probably has been the principal way of obtain- 
ing evaluations of research for.some time. It is no secret that scien- 
tists prefer this as a_method of. evaluating proposals rather than a 
quantitative-approach such as that you have taken: --- - 


_ We have received testimony from.other people which haa sug- 
gested that we should look -at a researcher’s track record..What 


have they published? What is his track record as it relates to this 


Professor MARTIN. Certainly there is, or was, a lot of skepticism, 
as I say, in the early days when we were doing this, and not just 
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How much experience do you have in those fields before you 
evaluate an individual researcher? == Bee Sees 
_.Professor-Martin. In-none of our studies have we focused on in- 
dividual researchers. The main reason-for_this is that _we-see the 
main problems from a science policymaking point of. view. focusing 
on large groups, large institutions, and so on. This is particularly 
true in the United Kingdom where our science budget, the money 
that-goes to the research council, is very heavily concentrated on a 
few laboratories. —-- - ee Se 


If you_take-the-biggest council, the Science and Engineering Re- 


search Council; about 60 or 70 percent of the money it spends on 
science goes-on-certain laboratories. So from a policymaking point 
of bets we think it’s most intriguing to focus on some of these big 
facilities. -- - : oe fe : : 


apply these sorts of indicators. to-individuals. There’s too much in- 


unless handled with very great care; in other words, with a lot of 
peer review. -- S64 62 = S, abs Re oe Be eS 
--Mr.-FuquA. It would be better with a larger group such as lab- 
oratories? = paar ae = Gas er eee eg eee ee an 2 oe 
_- Professor MartIN. It_is, because to some extent you.can. then 
choose-comparable groups with. for example, optical telescopes, you 
can look at similar-sized telescopes, with-accelerators; you can look 
at similar energy telescopes. So it’s possible to find matched groups 
that_you-can -apply these to, perhaps- more -accurately than one 
are with individuals where everybody's doing slightly different 
things... » 2 & 2) 2 
Mr. Fuqua. Mr.-Lujan? | 
Mr. L 


- It’s really quite interesting testimony: It will take.some time to 


think to our task force in studying science policy. . ~~ : 
- One thing that stands out as you go through your testimony is 
tnat certain laboratories, say in the accelerator business, rate hi 


fied that “we're the leader.” ae oe ee 
-. Now-in transportation, some transportation company will be the 
big one. In a bank, in a travel agency, real-estate agency—those 
kinds of service businesses, you notice one company for a period of 
time dominating the scene and then moving on. So it’s not so un- 


usual in science. - ae Be ee ee es 
=. Do you think that will continue and does the fact that someone 
makes a discovery just spur the other one to-a little more activity? 
Do ;ou-think studies are partially responsible for the lead moving 
from laboratory to laboratory? 
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They had_good-researchers; they-did some good projects which were 
very—had a big impact on: industry... Sores, ee ah, abies, (ee ee yee 
- However, over the last 10, 15 years one of those two laboratories 
has—the processor what. you-have just described has taken place; 


petence as. outsiders to.make judgments on.them. = aE ER aa OE, 
_ It’s interesting, though,;-that when I went back to Scandanavia 2 
or. 3 months—sorry, 2-or 3 years later to attend a conference, and 
we were a little bit worried when we saw the list of attendees, 
people attending the conference.-One of them was the director -of 


this institute, who had been rather rude to us at the end of the 
study. . —— z 

our report had some one gs one that-took him a little while to 
get used to—he then used the results of our report—and we were 
outsiders, we had no axe to grind in the Norwegian situation—to 
force through changes in his laboratory that would otherwise have 
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we spoke to him, or the year before perhaps, the laboratory-had 
been so successful it had made more profit from the work that it 


getting richer process, because if some new-university.or new labo- 
ratory has not had very good success, if-we follow your method of 
funding—you can-overcome that by considering the fact we have a 
big emphasis on training scientists. That is one-of the criteria that 
we use. How can. we overcome that, big laboratories getting all the 
at the little ones not? - a4 S 


where the poor, the poorly represented, the smaller. laboratories, 
because they are not well represented on decisionmaking bodies, 
don’t get their new ideas funded. And one good example of this was 
the SPEAR accelerator at SLAC: At the time, SLAC did not havea 
large-outside user group in the way that the two other big U-S. lab- 


oratories did, and this. may have been-one-reason why. Although 


that looked a very good project and subsequently turned. out. to be 
perhaps. the -most- successful. accelerator ever built; it was turned 
down on three occasions by the high-energy physics community at 


large. 


by interviewing researchers in the field, -again-with a-structured 


out, what new ones might emerge. And on_ the basis of that, I 
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pects for new accelerators in a way that does not necessarily lead 
to the rich getting richer... woh, Seee es (ees Nee ge ee See ee ee ane 
_ If I just give one example, one-of the laboratories that we looked 
at_ isthe. German--National Laboratory, DESY, which is- by no 
means one of the largest accelerator laboratories in the world. Yet, 
the machine that they’re proposing tc build; an electron-proton col- 
lider, looks, on the basis of the criteria-which we drew up as-to fac- 
tors that are likely to structure. performance in- the future, as 
though-it-might.be potentially the most interesting machine to 
build at the present. 7 zs a Sisto =). S. Se pete ae Oe, Ode eee ee eee en 
- So I would suggest that; if you look at factors structuring per- 
formance, one-would get around this tendency to just allow the 
rich to get richer; — - 7 ae . : oe 
Mr. LugaNn. One final question—really two questions which are 


use, to a greater or_lesser extent, some of these quantitative indica- 


particular_of .the very interesting work that is being done by Dr. 
Ho 


performi..ce. I’m less aware of work looking at institutional per- 
formance. - a Beene, Cee Se | eye ee See a ee ee ee 
_. Mr. Lusan. What kind of staffing—assuming you want: to evalu- 


of the main problems is insuring that you have complete publica- 
tion lists; and in many cases laboratories, for one reason or an- 
other, ° - ’t have-a complete list and we have to try to devise ways 


to make those lists more complete: 
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very good data to begin. with on publications. - 


r, UUIAN. 1U wot easier if you-tock one field? Say 
ies wanted to know who has done what or-the_primary work or 
it ye you really gotten results in, say, fusion, for example. 
Professor Martin. Yes.- =e Be) Sroae ee. See = a ee a2 = 1s = 
~-Mr. Lugan. That Me a much easier task than the type of 


-- Professor: - Yes. I’m just trying to think of other-examples 
that. might help put figures on this. At present the Royal Society in 
the United Kingdom is carrying out.a study on the health of Brit- 
ish science. They realized at the start there_was no way that they 


(o)) 
Mr. Fuqua. Mr. Packard? - ee stot, 
Mr. Packarp. I have no questions, Mr. Chairman. Thank you. 
_-Mr.- Fuqua. Professor Martin, based on your form of analysis, 
you arrived at any conclusions-concerning what government 
policies-tend to yield the best scientific results? That is very much 
part of the study-we are doing. - 


Professor Martin. Yes. No, that’s a-much broader issue than we 


have attempted to look at in any of our work: We, as I tried to ex- 
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proaches, and, therefore, over the 7 years that we’ve been doing 
this: work- there’s been a gradual and quite considerable evolution 
in the methods-that-we have been using. - Ss : : 


- It’s only in the last: year-that we've. begun to look at a- broad 


@-----  ----e----  -- = 


Councils..--. 2 2 Sono Mee a a ee eee ee ee ee eS Bg as eee ave Biase 
_ Mr. Fuqua. Your work has been; I gather; largely entered—or 
you related to it in your paper—on large laboratories. 

Professor MARTIN. Yes. : - Se 


__ Mr. Fuqua. How about evaluating -colleges and universities, and 


particularly departments? The specific question _I’m_interested_in 


is: we think that that is the resource whereby scientists are trained 
and is very important; that without the colleges and universities, 
then. we-would not have scientists working at Fermilab, CERN, or 
other places. - = 

Professor MARTIN. Yes.-- -_- eee 


_ Mr. Fuqua. How do-you take into account the training of future 
scientists not versus, but in addition to scientific papers and sci- 
ence that is produced there? _— =a 


~ Professor Martin. OK. Well, first of all, you’re absolutely right— 
it is important to look at. university and college departments, and 


in fact that’s, hopefully, the next project which we may be doing in 
the. coming mont ase, ae aes : : ae ne 
--As-I. explained,-we- decided, when -we -were. beginning our work 


seven years ago, to focus on the big laboratories because the British 
science is 80 heavily concentrated on those. 
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that. we will-find they work-relatively well for areas of more basic 


about it? Sie. Yee” Geese & = e z bc em Ze tees = - 
So one will need different indicators for different types of sci- 


we have attempted, in at -least.one.case,-to look -at the training 
function associated with-university research, and that was, again, 
the very first study -we.did-of radio astronomy, where we were in- 
terested in the people who had done radio astronomy as-postgradu- 
ate students. They had done master’s or they had done Ph.D.’s. 
Most of those-are_no longer-radio astronomers. There’s a limited 
job market for radio astronomers. Most of them have, therefore, 
moved on into a variety of positions. = 8 8 -§ — = 2 =: = 
-- And what we did was carry out.a-survey of, as far as-possible, all 


follow Prone ti:rough: successive: stages-of their careers.-As I say, 
th ; 


_And, three, in those jobs,which are the skills that you developed 


ture of the flows of trained personnel from radio astronomy into. a 


variety of jobs. A lot, for example; are now working in the telecom- 
munications industry, which is, as you know, a growing and impor- 
tant area of industry: a : es ae 
- And, secondly,-we were able to see the corts of skills which they 


developed as students which had been mot useful in the jobs that 
they had moved into. - dR ate ee —_ 
- So I would see-similar-sorts. of approaches for other areas per- 
haps generating the sorts of: information related to this training 
function which you mentioned. _ 

Mr. Fuqua. Anything further? 
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[No. response]. - 
..Professor ‘Martin, foak3 you: very ae for being iiers this morn- 
ing. It has been very interesting and. you-have a very unique ap- 
proach to evaluation. We-appreciate your input: 
Professor. MarTIN: Thank you very much. 
upon, at 9:40 a.m.; the task force recessed to reconvene at 


{Where 
the call of the Chair.] 
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Internal Criteria for Scientific Choice: 
An Evaluation of Research in 
__ High- Energy Physics 


BENJAMIN R. MARTIN AND JOHN IRVINE 


THe economic situation. of scientific research is now very different from 
what it was in the early !960s.-when -Dr. Alvin Weinberg opened -the 
debate on the criteria for scientific choice.’ Annual-rates of growth of 10 
per cent. or more in the budget for science were then common in most 
Western countries, while. today scientists face the prospect of no growth 
at all or even a decline. Some progress has also been made in developing 
techniques for the evaluation of the scientific performance of research 
groups. These two facts make it interesting to reconsider the question of 
Scientific choice. 


The Need a Evaluation i in Large-scale Research 


As recently as 50 years ago, basic scientific research _was s still.a relatively 


inexpensive activity. Since.the Second World War, however, the costs of 


research have soared, even allowing for inflation. During this period, 


science has flourished, with numerous new specialities starting, growing 


and maturing, many based on technological innovations originally created 


to meet industrial or aU, eds Prominent — new. special 


of millions of dollars. With this dramatic_ change i in. both the nature and 


scale of research activity, the need for procedures and criteria which will 


help in determining the choices necessary for the conduct of science policy 


has become all the greater. When the sums of money expended on_ basic 


scientific research were relatively small, making these choices was not 


such an urgent concern, except to the scientists themselves. However, as 


research has grown. more costly, it_has become all the more important 


that scientific resources should be used as effectively as possible. As Dr. 


Weinberg pointed out: _ 
It is only. when-science-really | does fake serious demands on our sciety—when 


it becomes 5 “Sig Science” ee es hqnese of cane ce really arises.’ 


Mass.: MiT Press, Saat en OMIA, 
7 Weinberg. A.M.. op. cu., p. 171 and Shils, E. (ed.). op. cit. p. 33. 
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effectively constricted only onthe basis of detailed jatoaination concerning 
the Specific results achieved in different areas of research; -and on the 


research groups. To facilitate the-collection and appraisal: of such infor- 
mation, scientists must be-more willing to-have the-results of their research 
scrutinised by other scientists, especially those working in different special- 
ities, as well as by the wider public. Scientific autonomy, inthe sense of 
the freedom of scientists to decide how the resources for science: should 
be. distributed, must eventually be reconciled with considerations. of 
efficiency when the eXpenentTS involved exceed a: certain Critical -size. 

For the “big sciences”, that point has been reached. Dr. Weinberg, 
himself a physicist, recognises that: 

It ts is. as much out of. a ‘prudent concer for: their own survival, as. for any Joftier 


broader point of view than -has: been iheir -custom. To do less would cause 3 
popular reaction which would damage mankind's most remarkable intellectual 


attainment—modem science—and the scientists who created it and must carry it 
forward.” aa 


construction of more effective science. salle but also to ensure that, in 
an. era of growing. demands for accountability, the public is provided. with 


the necessary assurances that government funds expended on research 
have been. employed efficiently i in the past, and i are likely to continue to 


be_so employed in the future. 
_ Four ‘main questions | need to be considered i in any attempt to construct 


scientific research compared with other objects: of public expenditure? 
How should this sum be distributed-among the different disciplines; with 
their competing claims for funds? How much should be allocated to the 
different types of scientific institutions—aur.iversity departments, regional 
or national laboratories, and international centres?-What resources should 
be given to each research centre, group, or individual working within i a 
discipline? _ 

- All these questions were to-some extent discussed during the 19606 i in 
the-debate on criteria for scientific choice. To sender the making -of 
decisions on such questions more systematic, Dr. Weinberg proposed a 
number of criteria, distinguishing in particular between “ifiternal” and 

“external” criteria: 
Internal criteria are: generated within the scientific field itself and answer the 


question: How well: is the science-done?-External.criteria are generated: outside 


the scientific field and answer the question: Why. pursue this particular science“ 


Dr: Weinberg’ s ideas subsequently attracted much attention in the scien- 
tific community and among the makers of science policy. However, 


3 Weinberg. AM. 0p. et. p: 171 and Shils: E. (ed.): op: cit: p: xu 
f oe aj ied sua E tt oe p. 25. 
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participants in the debate concentrated rather more on the external than 
the internal criteria, many were interested primarily in the level. of 
expenditure on basic research, basing their arguments.as to what that 
level should be-either on the notion of an overhead. charge in applied 
research. 01 or on n the idea of basic science as.a part of “high | civilisation”. 


from that i in the 1960s. _The reallocation of governmental expenditures i in 


recent years means that science must. at least.in.the near-future, face the 


prospect of little.or-no growth at all.° Consequently, decisions concerning 


the distribution_of the-funds for-science are_beginning to assume greater 


importance than those concerned solely with the total es ais on 
basic research... = —_—— cease Sele os 


_ The onset of_a period. of stable. expenditure. for. science. clearly. makes 


the problen: of determining choices within science more acute. If resources 


are to. be. made available forthe support of promising new fields_of 


science. this. can_ ae done _by_ pecans from_old,. feel 


ne nature. can. be based i in part_on the use of_ external criteria, it must 


also take account of internal factors: we should ask which fields of science 


and which. research groups are making or are eae to make the greater 


contributions_to science: _______ : 


_. However, any attempt to redistribute r resources is ; likely to encounter 


many difficulties. The specialities and groups that have vec’ "ed substantial 


that allocate funds for research: Because the. representation of scientists 


on committees allocating funds is generally in. Proportion to prevailing 


levels of scientific support, “organisational inertia” may act against change 


in the distribution of resources: As a result; certain specialities « or research 


groups. may still receive relatively large sums despite a decline in their 


scientific achievements. : 

_In some. circumstances, it may well be r necessary to cease supporting 
certain research activities altogether in order to release resources that can 
be employed more fruitfully in other areas of science. The desire to’ 
protect. individual or institutional interests may, however, impede the 


redirection of resources towards promising new areas of science or towards 


there is a danger that the internal regulation of research by the judeenicnt 
of peers will be unable to cope with the political pressures hindering the 
shift of resources towards more meritorious recipients. This aanget is 
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greater. where research ina. field i is concentrated in a few centres and 


where the judgements .as to the allocation of funds are made by panels of 


members_drawn largely from the users of those few centres. In such a 


situation there are not enough “neutral peers”. For this reason, a supple- 


mentary. Or alternative mode. of assessment of scientific achievements 


should_be considered. Such an assessment can be based in part on tools 


developed over the past few years within the “science of science”. 


The Quantitative Assessment of Scientific Achievement noe 


_ Many of the limitations of measures such as the- numbers. of research 


publications, or the numbers of references to those publications, have 


been ignored in their early, often. indiscriminate, use i in- scientific ; assess- 


which i is more realistic. It i is based on a number of converging “partial 


indicators”. -- . ees 
-We-have ‘pollen this method to evaluate. the. achievements of a group 


Seog Se ee 


of high-energy physicists who used the major British. electron. accelerator, 


NINA. operated by the Daresbury Laboraiory.. This accelerator_was 
closed-in 1977, a decision which met with considerable. Seo Soe 


mation put: forward to ae the decision. The arguments over the future 
of the accelerator emphasized its costliness rather than being based on a 


detailed evaluation of the contributions to scientific spas 6 made by 
the. physicists ‘tsing 1 the accelerator. 


From the late 1950s to the mid-1970s, this. field -of physics: annually 


accounted for about 40 Per cent. or more of the British. basic science 


appropriation for = hig eneas sles was-used to; pay Britain's s contribu- 
tion to the European Organization for Nuclear Research (CERN); the 
remainder was largely accounted for by the two national high-energy 
phisics. centres at the Rutherford and the Daresbury Laboratories, with 
a relatively small amount for direct grants to snidivichaal Scientists at 


-? in 1963-64 totat sapentiiis “on ihe National Institute for-Research in Nuclear Science: (NIRNS}— 


the body responsible for the operation of Daresbury Laboratory as well_as the other national accelerator 
centre: at Rutherford_Laboratory——amounted : to -£8:4 million. To this_must be added the British 
contribution of £2 million to CERN and-a major part of the £1 million in research grants allocated to 
universitics for nucicar_ physics research by the Department of Scientific and Industrial Research (DSIR): 
making a total-of about £11-million. In contrast, DSIR gave grants of £2-7 million to the Universities (0 
support all other fickis of scientific research. and £2 million for awards for | postgraduate training: In 
addition, DSIR was respomible for £1-4 million expenditure on British contributions to international 


Organisations other than CERN. -yickding a total of £6-t million which is only slightly more than half the 
£11-4 millon spent on high-energy physics. 
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universities. This high -degree of concentration of resources on three 
centres. within one field of physics greatly affected the scale of activity 


‘throughout, the rest of British basic science. 


- The Daresbury Laboratory was established i in. 1963 to operate a4GeV 
electron accelerator, ee Serving university physicists in the north of 


average -to some. 8 or 9 pe cent. of the annual budget of-the Science 
Research Council.’ : "and, byt the time NINA was. closed in 1977, a total of 


In assessing the correctness of the decision to discontinue ‘he operation 
of NINA-a number of. questions are-pertinent.. The main. question is: 
exactly what returns did this expenditure of £40 million brig? More 
specifically: what contributions to scientific-knowledge did NINA make? 
The construction of NINA had been authorised in order to enable British 
scientists to undertake “the investigation of quite new fundamenial prob- 
lems in the physics of elementary particles”."" To what extent did it 
contribute to the solution of “tundamental problems”? 


The Assessiient of Research Groups te ht ee oe 
_ Evaluations of the achievements of scientists engaged i in n basic research 


have generally been concentrated. onthe. individual scientists. or the 


speciality as a whole, rather_than the research group. This is not always 


a helpful approach since most-scientific resources tend to be allocated 


directly to research groups rather than to individuals or to specialities." 


The products of scientific research_can take_a variety of forms such as 


contributions to. knowledge in the discipline concerned _or_in neighbouring 


disciplines, or. the training of_scientists; they_can also take the form. of 


contributions to industry in terms of technological. knowledge and skilled 


persons. Because of the diverse and somewhat intangible ‘nature of its 


products, no comprehensive. and. unitary. quantification of. the.achieve- 


ments of basic science is possible: In view of this, assessment. of the 


performance of a-scientific group must be based ona comparison of the 


achievements of one group with those of others utilising broadly similar 


amounts of money and working in the same scientific field. We can, of 


* The research of high-cnesgy physicists at southern British Universities was ‘conducted raily at 


Rutherford Laboratory or at-CERN. - 
“Dd 


stator. it was Bp cd cent. - although 


976-77. the last ‘year of Operation cf the 
-high-cnergy rh 


'F 
11.M. Stationery Office. 1962). p. 142 : 

" Sce Irvine. J. and Martin.. B.! R:. cA Methodology for hai the Scientific Performance of 
Research Groups”. in Proceedings of the Conference On Evaluation of Science and Technology: Theory 
and Practice. Dubrovnik. Yugoslavia. July 1980, published in Scientia Yugostavica, 6 (April 1980). pp. 
Hv-95, 
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course, legitimately compare only “ke” with “like”: It would not be 


reasonable, for example, to compare the scientific achievements, in terms 


of numbers of discoveries or costs per paper, of a group of high-energy 


physicists using an accelerator like NINA with those of a group of 
astronomers using a large telescope. seeders ee re ee, cee! | 
Since _no two groups produce identical_kinds of results, the task of 


selecting exactly matched groups for comparison might appear to be a 
virtual impossibility. Even two. groups working in the same field and 
producing fairly similar results may have very different educational and 
technological effects. In high-energy physics, however, a certain degree of 
simplification is justified. High-energy physics is; above all, a basic science. 
the primary purpose of which is to increase scientific knowledge. rather 
than create educational, technological, or cultural benefits. In arriving at 


decisions on the distribution of resources within basic science, the main 
criterion must therefore be that of how great a contribution is likely to be 
made to scientific knowledge. This was Dr. Weinberg’s fundamental 
internal criterion. --- --- — - - : : Sy eee es 

In the case of British high-energy physics, there have undoubtedly been 
external benefits in the form of technological “spin-off” and_trained 
research workers for industry,* but.these were not the primary reason for 
supporting research in this field.'S Important external benefits can be no 
more than supplementary reasons to-support scientific activities that are 


Is the field ripe for exploitation? And are-the scientists in the-field really 
competent?'® He adds that; in practice, the second of these is normally 
the main. one used in assessing applications for-research grants. The 
question implied by the second.criterion can best be. answered by ascer- 
taining how much the work. of the group of scientists in question has 
contributed to.the advance of knowledge. Past achievement; of course, is 
not the only clue-to future performance, but it-is undeniably one of the 
most important. If-other. factors.such as the “ripeness” of the proposed 
research topic or its potential for scientific exploitation are equal; “OF 
indeterminate, a new scientific project-clearly stands more chance of being 
carried out successfully -by a -group with a record of major contributions 
to the advancement of knowledge over the recent past than by one with © 
an undistinguished record. ; 

"See Toulmin,S:, “The Complexity of Scientific Choice: A Stockiaking”. Minerva. 11 (Spring 1964), 
p:.354 and Shits, E. (€d.). op. cit., pp. 63-79. OO ee ge ee ey a eee 
__'* Some of the more important economic benefits to firms are pointed to in Schmicd. H.. “A Study of 
Economic Utility from CERN Contracts", IEEE Transactions on Engineering Management, EM-XX1V 
(1977), pp. 125-138. - -- be ae Pe ee ee ee eee 
-.'* However, the probability of technological by-products has been advanced as a reason for supporting 
very high-cost basic science.See Richter. B.. “Statement before the House Committce on Science and 
Techaulogy, December 10.1979" (Stanford Linear Accelerator Center. Stanford University) mimeograph, 
for. a-revicw of-some of the. more. important developments. Even in this case. such “spin-off” would 
hardly constitute. a justification for Supporting Mediocre fesearch, 

“ Weinberg, A. M.. op. cit.. p. 163 and Shils, E. (ed.), op. cit.. p. 2S. 
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Dr. Weinberg’s. first internal criterion. 1 should p perhaps be reformulated 


in two_parts: What is the scate of the field itself—is it ripe for exploitation? 


And _how-well_placed is the applicant-group in. comparison. with its 


probable..competitors—does_ it possess some advantage enabling it_to 
exploit the. field_and to. achieve Significant Scientific results better. than 


advance. it is difficult to se see. how one can avoid depending o on the opinions 


the proponents. of the. research Project who, presumably, would r not. have 


advancement: and, on n the other, those scientists who are engaged i in rival 


activities within the ficld; and who are to some extent in competition with 


the project's proponents. Hence, the opinions of neither body of scientists 
1r..1y-be unqualifiedly neutral and objective. Even scientists in neighbouring 


fields, who constitute a third group of possible referees, may also make 


recommendations which are_at least partly influenced "y their ceare | to 


assure money for their own research. _ 
_ The. a of the Saag to exploit a field ahead of its 


Over the recent past -then the 5 group applying for. support cannot generally 


be regarded as being in the best position to exploit the: scientific poten- 


tialities of that field: The only exception to this is where a highly innovative 


research instrument is proposed which-would place an entirely new group, 


or a group that had not done well in the past, in a unique Position within 


its field. The procedure which we propose cannot determine the “ripeness” 


of a field for “exploitation”; at present only the judgement: pf qualified 


scientists can offer guidance here. Once such a decision is madé, however, 


officials charged with judging applications for grants should pay attention 


to.the relative merits of the applicants as ascertained by a scrutiny of their 


respective previo us achievements. 


Indicators of Scientific Performance 

Since. most research results are published . in journals or conference 
proceedings. one obvious way to assess scientific performance is by the 
number of papers published by a scientist or group."” But all papers 3 are 
not equal contributions to science. Data on the “numbers of papers” can 
be combined with data on the financial resources needed to produce these 
Papers to yield such figures as the “cost per publication” and the “publi- 
cations per research worker per year". This does not necessarily tell cs 
anything about a group's relative contribution to scientific knowledge, 


"7 Far a summary of some investigations along these lines, sce Githert, G. N. and Woolgar, 3%, “The 
Onaniinative Study of Scicnce: An Examination of the ratice”, £° nce Studi: + (July i974). pp. 
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powered Practices in 1 publication. vary. considerably between. research 


groups; differences in the pressure to publish in order to_enhance the 


prospects of promotion or of future_support may_result in_differences.in 


the rate of publication. Furthermore, one group. may conduct_several 


sheic. research projects, — many Papers, while apes engages in 


fiumber of publications? One is the number of times the published work 


of a research group is cited by other scientists. Citations have been used 


as a-measure of the “quality” of scientific research in much previous work 


on this problem. There are; however, many difficulties in this procedure. 


The Science Citation Index, which is generally used in these studies, does 


not provide a sufficiently complete record of citations. For example, 
citations are credited only to the first-named author of papers with more 
than one-author; there are variations in authors’ names (for exampie; in 
the use of one rather than two initials); there may be several authors with 
identical names; arJ finally there are numerous errors. arising from the 
failure of authors to-give the correct bibliographical references. The first 
of the difficulties is less severe when we deal with a group of scientists, 
since all citations are-credited to the group and not to individuals, while 
many of the other problems can be largely overcome by careful checking.'" 
- Despite this, serious difficulties in the use of citations remain. For one 
= certain es Papers gels go ees in the scientific 


the. -body of scientific knowledge that the original ane cvertually cease 
to be cited. In addition, some papers may be referred to critically-— - 
because they are viewed as “mistaken”—while others may be cited purely 
because the prominence of their authors is thought to add authority to an 
argument.” However, if the number of citations to a paper is regarded as 
an indication of its impact on the advancement of scientific knowledge 
rather than its intrinsic scientific quality or importance, thea many of the 
problems Of citation-analysis become less severe. __ 

-Here, we define the “quality” of a research -paper-in terms of the 
absence -Of obvious. “error”; the aesthetic merit of the mathematical 
formulations, the originality of the conclusions; and so on. The “import- 


es See Martin, B. R. and Irvine. J.. "Absesaing:| Basic Research: Some Partial Indicators of Scientific 
Progress i in R. Astronomy”, Research Policy (forthcoming). 
in i computer tapes on | Citations available from the Institute for 
Scic ti the Science Citation index. - - 

» F. E.. “The Measurement of Scholarly Work: A Critical Review of the 
Literature”. Educational Record (Summer_1971). pp. 225-232. These and ather problems in citation: 
analysis are discussed at length in Martin. B. R. and irvine. J.. op. cit. 
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ance™ and “impact” of a research paper, i contrast; feteriene relations 
between the scientific results reported and -current- work in associated 
areas of research. The “importance” of a publication describes its potential 
influence on neighbouring research. activities—that -is, the. influence it 
would. have on the advancement of. scientific knowledge if there were 
perfect. communication in science. However, there are imperfections in 
the system of scientific communications; for example, a paper by a young 
Scientist at a provincial university, published in a journal in a language 
other than English and-having a small circulation, may go unnoticed by 
scientists regardless of its merits. Therefore, the “importance” of a paper 
is generally not synonymous with its “impact”. It is-the “impact” of a 
paper which represents. its actual influence at a particular time on sur- 
rounding research activities and -hence on the- advancement of 
knowledge,”! and it is this impact, rather than its quality or importance, 
that is-reflected in the number. of citations earned by a particular paper.-- 
- While. the impact of the publishcd work of a group or research unit will 
depend to a large extent on the work's importance, it is also influenced by 
such factors as the status of the institution in which the group is working, 
and the prestige, language and circulation of the journals in which- its 
results are published. The number of citations received by a publication 
is an indicator of its impact. However, it is by no means a perfect measure 
of impact—the frequency of citation may vary with type of paper and 
with speciality, which is why citations can only be used to compare 
matched groups producing sir::!ar types of papers-within the same special- 
ity. Since it-is the impact. rather than. the-iniportance of the publication 
which reflects its contribution to scientific knc wledge, we may regard the 
number of citations as a partial indicator of the scientific achievements of 
a research group. +. 

_ The method of assessing contributions to scientific knowiddge generally 
favoured by scientists is “assessment by peers”. However, again this is not 
a wholly adequate measure of scientific achievement. Any assessment 
made by scientists of the-work of their peers is subj ‘o influence by 
social and. political considerations within the scientifi. community. Few 
scientists are unaware of the fact that such assessments may have reper- 


cussions.on the future support-of both their own group and their competi- 


tors. Furthermore, individual scientists can only assess the scientific 


contributions of others in relation to their own cognitive locations,” and 


assessments.may differ. Finally, scientists tend to conform to convention- 


ally accepted. assessments; they may feel it necessary to praise publicly the 


work of eminent scientists, even if privately they_hold that. work-in_low 


esteem; they might also be ignorant of the work of a particular research 


=: 9 One of Dr: Weinherg’s internal criteria: “How well is the scicnce done?” should theretore pethags 
he. telonmatated as. “How. much impact_on the advance of knowledge does the rescarch have?" 

2 See Martin. B. R.. “Cognitive and Sicial_ Locations: Their fein the Processes of Discovery. 
Evatuation within Science”. (Department of Liberal Studies in . University of Manchester: feos 
mimcograph. 
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group and might therefore make assessments more on the general repu- 


expressed and actual views held, the scientists being interviewed could be 
pressed further. Even so, it is clear that assessment by. peers, like citations 


and. publications, gives no more than a partial indication of the scientific 
contributions of research. groups. ee ree 
-- In assessing the scientific achievements of the group working with NINA 


relative.to_those of groups of scientists working with other accelerators, 
we have therefore employed indicators based-on publications, citations 
and _peer-evaluation. By counting the total number of experimental papers 


each research centre. We considered both the average number of citations 
per paper for the papers produced by each group (the “citation-rate”) and 
the number of very highly cited papers, The former provides an indicator 
of the average influence of all the papers produced by the users of each 


made. not only in the form of gradual change—to which all publications 
may be said to contribute—but also through the occasional, revolutionary 
transformations resulting from only a small number of major papers. A 


responsible for one.or two major. discoveries, reported in papers which 
have been cited with especially high frequency. Assessment by peers 
complements these two measures, providing some soit of weighted average 
indicator for both the great mass of relatively minor contributions made 


We have attempted to compare the contributions to scientific knowledge 
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make possible direct comparisons. of results: the Cambridge Electron 


Accelerator {CEA) operated by Harvard University and Massachusetts 
Institute of Technology;-the Deutsches Elektronen Synchrotron (DESY) 
at Hamburg; anda Russian machine.at Yerevan-in Armenia, although we 
have very little data-on this, apart from some judgements by individual 
scientists. The energies of the four accelerators ranged between approxi- 
mately 4-and 6 GeV,.and all were completed-in the: 1960s. The only other 
major electron accelerator built-during this period- was the 20 GeV 
Stanford Linear Accelerator (SLAC), and this was also included in our 
Study (Figure I). 


Ficune I 
; The World's s Main Electron Accelerators _ 
Centre Details of Accelerator(s) Completion 
Zs = e.. Joe : 2S. wSee See s = = . date 
Cambridge Electron 6 GeV Electron Synchrotron 1962 
Accelerator (CEA) BYPASS—3-5 GeV Electron- hae 
Positron Storage Ring 1972 
Daresbury NINA=4'5 GeV Electron — 
Laboratory Synchrotron 1966 
Deutsches. 6 GeV Electron Synchrotron* 1964 
Elektronen DORIS—3 GeV Electron- 

Synchrotron, Positron Storage Ring’ __ 1973 

Hamburg (DESY) PETRA—19 GeV Electron- 
Positron Storage Ring 1978 
Stanford Linear 20 GeV Linear Electron . ee 
Accelerator — - Accelerator _- - — 1966 
Center (SLAC) 2 SPEAR--2-5 GeV. Electron- shoe 
Positron Storage King 1972 
PEP—18 GeV Electron- aaa 
Positron Storage Ring 1980 
Yerevan 6 GeV Electrop Sgnckrotton 1967 


: Increased to 7. SGeV i in 1958. 
* Increased to S GeV in 1976. 


Source: Annual reports from the centres. 


Measures oft the Scale of Research at Each Accelerator - 


_Before.examining the various _indicators_of scientific achievement, ‘itis ; 


first necessary. to_consider the scale of research activity. carried on at each 


accelerator; in pesticuiat; the number of scientists doing research at each 
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TABLE i 7 
Numbers of High-Energy Physicists using the Four Centres 
1966-1978 

_CEA. DESY NINA. SLAC 
HSB PS ee (Cambridge) (Hamburg) (Daresbury) (Stanford) 
-..- Central research staff. 19 ~o ~1SS OF 
1966 External experimental users 4 ~25 ~3F = ~4r 


Research students 6 ~15 ~2 ~2F 
Total 125 ~100 ~65 ~120° 


1970 External experimental users 23 ~50 


1974 External experimental users 12 ~70 


1978 Extemal experimental users ~130 ~200 


’ 
3 

BRS a, SSRS SRR): 
l 
3 


=: * This excludes synchrotron-radiation research workers. But includes high-energy theoretical physicists 
resident at cach of the centres, since theit_research is generally so closely related to the experimental 
work as to constitute an integral part Of it. The errors in these figures should be no greater than 10% or 


1S% st most. 


of the. accelerators has varied considerably both among the centres and 


over time (Table I) = 6 eanens sane nein mance eienie | 

- Almost from the time the Stanford Linear Accelerator began operation, 
it has-supported the largest-research group of the four. By 1970, it. was 
providing experimental facilities for over twice. as many physicists as the 
Daresbury synchrotron, and about 30 per cent. more than the. German 
synchrotron at Hamburg. The group using the Cambridge accelerator was 
by this time decreasing in size, largely as a result_of.a major-explosion in 
the late 1960s which seriously restricted its research programme; by 1970, 
it provided experimental facilities for only approximately three fourths of 
the number using- NINA .at Daresbury. Up. to 1974, the ratio of the 
numbers of physicists at Hamburg and at Stauford stayed at about. 2:3; 


both centres had grown rapidly, so that by this time the number of 
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research workers using the Hamburg centre was three times that using 
Daresbury. In the meantime, the centre at Cambridge had closed_because 
of the American policy of concentration of high-energy physics activity in 
three major centres. Between 1974 and 1978, the number of users_of the 
Hamburg centre contiaved to grow, expanding by more than 50 per cent. 
tO approach the size of the group_using the Stanford. accelerator.In 


was closed in 1977. 


- In estimating the scale of research activity associated with an accelera- 
tor, it is necessary to bear in mind that not all users were able to spend 
all their.time on research; some of them had other tasks as well, not least 


the teaching of undergraduates.** The external users in particular had to 


spend substantial fractions of their time on teaching (Table II). 
Taste I 
Numbers of “Effecsive” High-Energy Physicists using Eack Accelerator 
1966-1978 re 
a EA. DESY NINA SLAC 
Percentage of time spent in teaching” =~ 


Central research staff- S% SF 8% 
Go I% 25 


Total number of “effective” research 


1966 115 ~# as ~10F 
1970 62. ~130 73 ~i8? 
1974 2P ~18§ 48 | ~255 
1978 — ~255 279! ~=305 
_* Incalculating the “cffective” number of physicists. we fiave assumed that the percentages reported 


in 1979-80 did not change sj-preciahly over the preceding 10-15 years. 


* 1972. See Table 1, note b. 
© 1967. See Table 1. note ec. 
_ © 1976-77. See Tabic 1, note d. - 


Souacé: Arinual reports from the centres and private correspondence. 


Another. major factor determining the-scale_of research activity is 
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Tape I! 
Aviat Operating Costs (in pounds sterling)> 


= tees See aes Te - CEA DESY NINA‘ SLAC 
Annual scaring costs (in millions) 


1966 £13 £3-4 £2-7' _£6-4! 
1970 £13 £49. £3-9 £10-0 
1974 £0-9 £109 £3-2 £10-2 
1978 —.  £20:3° £1-8* £15-5 


Annual. c cost t per. physicist (including 


students) (in thousands) in: 


1966 £10 ~£35 £25'  ~£55' 
1970 £220 ~85 5 ~£50 
1974 £30° ~£55 £55 ~£35 
1978 = ~£70 £508 ~£55 

; Operating costs are exclusive of salaries of visiting Phi, which are i br ¢ their horse 
institutiogs. 


© See Table I. noteb.  — ats 


é These figures have been calculated by adjusting the actual total operating cout to allow forthe effort put ito 
irecprowron reiaion work (espeoimately DM 1 sition per seo). st a 


Sencha son ec eee Pl PE 


operating costs for the Daresbury synchrotron were approximately three 


times those of the Cambridge machine. The next most expensive was the 
German machine. By 1974 its operating costs had grown so rapidly that 
they were similar to those of the Stanford Linear Accelerator, both being 
just over three times larger than those of NINA. During the most recent 
four-year period, the costs of the Hamburg and the Stanford establish- 
ments have both almost. doubled as-new soragecring facilities were built 
_ As with Operating costs, capital costs are not strictly comparable, again 
because of differences in accounting procedures among centres; there 
were, for example, differences over whether-or not labour costs internal 
to the-centre, or ths costs of certain buildings, shou'd be included as 
expenditure on capital-equipment. Despite this, it can be seen-that; of the 
our Seba stietes the Cambridge emebretton was 2. far the least He coe 


Be eben was six times that of the frei synchrotron. The Stanford 
figure is particularly. high because it contains.a substantial amount. for 
“salaries”. We therefore calculated figures for the total expenditure, i.e. ; 
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TABLE IV oe 

Energy and Capital Costs of Original Accelerators* 
— CEA DESY NINA SLAC 
Energey .§.§ |_| _ 6GeV 6 GeV 4 GeV 20 GeV 
Quoted _capital. cost” (in millions of 22.5 eS ies sees 
pcands sterling) a £1:9 £67 £4-4 £40 
Date of first experimental publication 1963 1965 1969 1967 
Total expenditure up to year of first 
publication (in millions of pounds i's = ee 
sterling) £5-5 £13-7 £148 £66 


results were published (Table IV).-The costs for the Hamburg synchrotron 
and NINA were very similar, while the Stanford accelerator was consider- 
ably more expensive. Its costs were four times greater; this is consistent 
with the ratio of the energies of the machic«'s. pote eee 
" Ofie final factor that influences the *cale and productivity of research 


centre had about 70 per.cent._more supporii-, swor ikan Da-esbury; at 
each, the number grew. by about 30 per cent. over the next -fotir years. 
During the 1970s, however, the number of supporting staff for users of 
the Daresbury accelerator decréased_as more effort was put into research 
on synchrotron radiation. and nuclear structure, while for Hamburg the 
number continued {> tise. Once the accelerators were..in-operation,- the 
ratio of supporting staff to physicists for Hamburg, NINA and Stanford 


dropped to about 4:1 in 1973; for INA it “‘ayed at about 6:1. 
If we consider ail these_measurcs together, they suggest that the scale 
of research activity at Stanford was initially considerably greater than at 
Hamburg, which was the largest of the other three centres. By the time 
of its first publication, the scale of research at the Daresbury synchrotron 
had overtaken the Cambridge facility in size, but it-was never more than 


about 60 per-cent. of that at the Hamburg centre. During the 1970s, the 


German centre_grew rapidly, with the construction. first of a. 3 Gev 


electron-positron storage ring and then a higher energy (19 GeV) storage 


ring (Figuze 1), so that by 1978 its level of activity was only perhaps 20 
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TABLE V 
Numbers of Supporting Staff-members* 


Numbers of stipporting staff-members 


1966 Central - 
User institutions 
Total 

1970 Central - 
User institutions 
Total 

1974 Central - - 
User institutions 


CEA DESY NINA SLAC 


180 690 


98 940 


~30 ~110 


~380° 1200° 
~40° ~80! 


) ~420% 1280° 


502 1200 
550 ~1340 


345 1000 
44 ~320 


Total ~128° ~1050 389 ~1320 


1978 Central. - — 950° 184 1100 
User institutions — 20 29% ~400 
Total — ~1150 215° ~1500 


Ratio of supporting staff to physicists: - - ne 
1966 20 7:3 6:54 10-78 
| KS 61 62 67 
1974 43 55 65 44 
1978 — 38 G6If 42 


® These include Ant only supporting staff empicyed at each centre. but also those at the home institutions of 
the visiting physicists. _ 

* See Table noteh eee 

© These are the figures obi-ined alter subtracting those staff engag:d in synchrotron-radistion work. 

4 See Table 1, not «. 

* See Table 1, note d. ; ; ; 
Source: Annual reports ire, che aenarcs and private corresponden 


per cent. smalle: than that of Stanford, which had built storage ring and 
was in the-course of constructing auother machirs 2):na<t identical to the 
German 19 GeV storage ring. 


Publications : : = aes : z S 2 bs 
H=ving obtained some idea of the respective scales of research activity 
at-e:ch accelerator, we.can now consider the first-of the indicators of 
scientific achievement—the number_of experimental publications arising 
from t..--i.se of each accelerator (Table VI). Over the first ten years of its 
life, the Hamburg synchrotron appears to have been relatively productive, 
research carried out there resulting in 50 per cent. more papers than was 
the case for the Cambridge Electron Accelerator (13 per annum compared 
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ee TAsLe VI 
Numbers of Published, Experimental High-Energy Physics Papers from 
Electron Accelerators (excluding Work on Storage Rings) 
Year © CEA DESY NINA SLAC 
1963 _ 


tot 
7) 
3 
o 
Stilt lt lunawabBsSiaan: 
— 
a 


— 

fo.) 
Riunanauaodsaan | | | | | | 

poe 

a 


Average no. -- +: - _ 

per year over 8 13 6 19 

first ten years ' oa - 
Source: Complete lists of publications supplied by the Daresbury, Hamburg and Stanford centres. 
Cambridge publications compiled from “Source Index” of Science Citation Index. 


with 8 per annum at Cambridge). The production of papers at the 


Cambridge. The linear accelerator at Stanford, having a greater number 
of users, produced far more publications than the accelerators at the other 
three centres; moreover, its rate of publication has actually continued to 
increase, more or less up to the present time_as new uses have been found 
for. the facility. Over its tirst ten years.of operation, the research workers 
at this-accelerator produced_on average about 50 per cent. more papers 
annually than those using the synchrotron at Hamburg, two and a half 
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Taste VII 


Average Numbers of Publications in Retation to Numbers of Users and 
Financial Resources* _ 
aa ae _ CEA DESY NINA SLAC 
No.- of papers per “effec- 
tive” physicist? in: eel 2. oo. oe Sy eS tat paar aan 
1966 6-13 0-18 0-01°* 0-10" 


1970 0-08 0-13 0-11 0-08 


1974 O-15* 0-08 0-10 0-13 
1978 | : _— 0-11 0-20! 0-12 
No. of papers per £m oper- 
ating costs? in: os : : 
1966 1-300 4-7 0-446 1.98 
1970 38 8-33 2-4 1-6 
1974 aa 12 16 38 
1978 = a2e =— ... 14 2-9! 2:2 

* These figures should be accurate tobetween 10 and ISpercent. == 
__* Because the number of papers published in any one year is fairly small, the random fluctuations from 


experiment end the publication of the results. Because the number of publications in 1980 were unknown 
at the time of writing. the 1978 figures are based on the average number of papers produced in 1978 and 
1979.- = 


“See Table I,notec. oe ae 2 ee BORE FaS 
se, although NINA was operating in 1967, the first publications did not appear 
age annual rate of publication for the period 1967 to 1969 is therefore low. 


Source: Cownplete ‘ists of publications supplied by the Daresbury. Hamburg and Stanford centres. 


lives as research. carried out .in- previous -years was written up and 
published, while the numbers of physicists still using the accelerators and 


the operating costs were. both falling rapidly. - a — 
As we argued earlier, however, it is not sufficient-to consider solely the 


numbers.of papers yielded by each accelerator. It is, for example, quite 


conceivable that each of the papers.from ore acceleratoi represented, on 
average, a more significant contribution to s-ientific knowledges than those 
from_the others. Unti! we have considered. the impact of those papers; we 


cannot reach any conclusions about the relative contributions to scientific 


progress made by the accelerators. 
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Aasessrnent of Impact — a 
_. Because of its. earlier beginning: the published work of the Cambridge 


centre was obviously the first to make an impact (Table Vill): By 1968, 


TABLE vill 
Citations of All Previous Electron-Accelerator Work (excluding Work o on 

_ Storage Rings) 7 oe 
Citations i in: CEA DESY NINA SLAC 
1964 28 - = = 
1968 29 3903 8B 
1970 160 306 19 405 
1972 149 358 72 504 
1974 103 324i its 506 
1976 30267654 
1978 76 305 57 585 
Average _no. ae = _- 
per year over 140 260 60 390 


_ first ten years — 
ounce: Manual analysis of data from Science Citation Index. 


however. it had been overtaken by the Hamburg synchrotron, ‘by. the 


years after the Cambridge centre had closed down. (in 1978, _however, 
papers originating from the Cambridge accelerator were more- frequently 
cited than those from NINA.) It was not long before the Stanfordicentre 
overtook Hamburg, generally receiving between 50 per.cent. and 100 per 


cent. more citations each year than the- German accelerator, which, in 


o__- 


satied out at Darebary, When the storape-tilig. work is ihelided, these 
differences are further-accentuated, with Stanford receiving four times as 
many citations as Mamie? in 1976 anid over twice as sia in 1978 (Table 


IX). 
-_. Whereas the total number of citations is to-a large extent t determined 


by the amount of research activity at each centre, the average number of 
citations per paper takes account of this factor, so that a small but 
relatively successful centre, even if it produces fewer papers, should still 
be able to achieve a relatively large number of citations per paper. The 


papers produced at Stanford none the- less appear to have had on average 


the greatest impact (Table X): In.contrast, NINA was.comparatively laie 


on the scene. By the time of its first. publications, many of the electron- 


synchrotron. experiments likely to have a-significant impact on scientific 


knowledge had already been carried out elsewhere. Moreover, even the 


76° 
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Tase Ix 
Citations 1 to » Ail Previous Experimental Work including Work on Storage 
nas __ Rings eee 
Citations in: CEA DESY NINA SLAC 
1964 -28 ed = = 
1966 139 Mo: = = 
1968 219 303 = 88 
1970 160 306 19 405 
1972 149 358 72 504 
1974 197 324 78 -512 
1976 67-295 61282 
1978 m:) 644 57 1463 
Source: Manual analysis of data from Science Citation Index, 
TABLE x 


Average Citations per Publication for Work Published i in inst Four Years 
(excluding Work on Storage Rings) 


Citations per_ CEA DESY NINA SLAC 
publication in: : 

1966 5-8 _ _ — 
1968 4-1 6-2 _— — 
1970 3-0 46 _ 8-1 
1972 4-0 4-4 28 6°5 
1974 3-9 3-2 2-3 49 
1976 _ 20 1:9 3-0 
1978 — "2S 0-9 2:7 


veaasins Mil analy of ata trom Since Con nde. 


best of these- early. electron - synchrotrons- probably. did fot t produce 
research results of the. highest importance for the advance of scientific 


_ knowledge. The storage-ring work at Stanford—in particular in-1976 when 


it-received -an. average of 25.citations per _paper—and at Hamburg (8-7 


citations per paper in 1978) had far greater impact than any of the 


research carried out onthe original synchrotrons (Table XI). This conclu- 


sion is in- tine. with the. view_of most. of the Physicists interviewed. that 


of storage. rings. Before this, elie accelerator physics had always been 


pre-eminent... _____- Se ee 


_Many. scientists argue that it is not the great bulk of papers that advance. 


scientific_knowledge most, but rather a small_number of papers. that 


exercise a “isproportionately large impact on the growth of knowledge. 


These. majo’. advances are, by and large, represented by very frequently 


cited papers (Table XII). The top one per cent. of most-cited publications 


a 
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Average Citations per Publication for All Work Published in ta:* Four 
Years Including Storage-Ring Work 


Ci itations 9¢ per. CEA DESY NINA SLAC 
publication in : 

1966 5-8 — _ — 

1968 4-1 6:2 _ _— 

1970 3-0 4-6 — 8-i 

1972 40 4-4 28 6-5 

1974 7-6 BL: 2:3 4-7. 

1976 _ "Bi (7: 2 19 8-5 (25: 1)" 
1978 | — 5-7 (8-7)* 0-9 6-1 (12-2)* 


: The figures i in brackets a are the citations per publication for he 2i+>2ge-ring work alone. 
Source: Manual analysis of data from Science Citation Index. 


_ Taste XII 
ae Numbers of Most Frequenily Cited a 
Numbers of oe Numbers of papers:* — 
eatin: CEA DESY NINA SLAC 
in one 
year St, Sy Seeees eye Nese 
12 19 (21) 24 (38) 1 33 (68) 
15 10 (12) 15 (26) 0 26 (58) 
20 4 (6) 4 (11) 0 12 (37) 
30 0 (2). 1 (4) 0 _ 3 (20) 
50 0 (1) 0 (0) 0 f 2 (10) 
* The Rates in brackets are numbers © of most frequently | cited 5 papers including those produced from 
work on storage rings. 


Source: Manual analysis of data from Science Citation Index. 


arising frem work on electron accelerators were cited 30 or.more times in 
one-year. Stanford users produced five.such papers, Hamburg one, and 
neither Cambridge nor NINA succeeded in producing any. Incleed, NINA 
did not produce. any papers cited 15 or more simes in any one year, while 
Cambridge produced ten, Hamburg iS and Stanford 26. 


Evaluation by Peers_ 


- We asked about 80 high-energy physicists to specify the. main contri- 
butions arising from-work on their own accelerator—or-on-the one which 
they used for most of their research—and on a number of others, including 
all the major proton and clectron high-energy physics accelerators. These 
same physicists were also requested to rank the accelerators in order. of 
the meri: of their contribiitions to science over the decade between 1969 
and 1978. 
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There « were slight differe erences between the physicists’. rankings. of work 


done « on their own accelerator and the rankings by colleagues elsewhere 


of-that work.” We regard the rankifigs made by the latter as the more 


valid indicator of the relative achievements of different accelerators: The 


high-energy physicists whom we interviewed, who, it should be stressed, 


were mainly electron high-energy physicists; believed the accelerators at 
Stanford contributed -most_tto scientific knowledge over the decade in 
question; they placed Stanford significantly ahead of the large. proton 


accelerator centres. at- CERN, _and- the two. other American national 


Bivokhiaven (Fable XIII). 
We also asked the high-energy phyaiciels to identify major changes i in 


TaBLe XIII 


Physicists’ Evaluations of the Contributions to Scientific Kiidiwiedes 
made by Accelerators at 13-High-Energy Physics Centres 


between 1969 and 1978 
(= highest ranking; 13=lowest ranking) : 
Evaluation by: or Average 
Physicists at: _ Other = -rankings 


oe high-energy (excluding 
Hamburg Daresbury Stanford physicists self-. 
(n=23) (n= 16) (n=16) (n=21) rankings) 


a Electron: eee ao See 
Cambridge 8-4 8: 1 Ps 1 8-6 8-3 
Hamburg 3-9 5-¢ 40 3-7 4:2 
Daresbury 73 7-0 8-5 8-1 8-0 
Stanford 17 1:8 13 16 17 
Yerevan 11-7 11-5 11-5 11-7 11-6 
- Proton:? © - - : 

Argonne 6°8 6-8 7:1 6-6 6:8 
Brookhaven 3:4 3:4 3-6 4:3 3:7 
CERN -3-0 3:1 2:4 2:5 2-7 
Dubna - 11-7 11-2 11-5 11-8 11-6 
Fermilab -3°5 2:7 3-8 3-0 3-2 
Moscow - 12-0 11-5 11-5 12:0 11-8 
Rutherford 8-2 9-2 8-3 76 8-3 
Serpukhov 9-5 9-8 O95 95 9-6 | 
= © This is a list. of the major ‘proton: accelerators (accelerators whose chictgies were are about 


7.GeV)-that were operating. during the ten-ycat period under Consideration. Three of these arc in the 
United Staies (Argonne. Brookhaven and Fermilah). three inthe Soviet Union (Dubna, Moscow and 
Serpakhov). one in Britain (Rotherford) and one on the Sui iss- -French border (CERN). 


* For. example, NINA ucts. gave themselves the ranking 7-0; whilc the average given by the other 
thrce groups of assessors was 8-0. 
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the relative. positions of these. centres < _over. r this period. In their view, at 


the beginning of the decade, the accelerators at_Brookhaven-and_perhaps 


Geneva (CERN) were. the. world-leaders -in high-energy. physics.. They 


thought thatthe leading position then passed to Stanford. largely because 


of the _experimental work carried out on its storage ring. For the major 


part of the _dccade, the world-leaders.in high-energy physics were con- 


sidered by_our reviewers to be the scientists working with the accelerators 


at Stanford. thz European_Organization. for. Nuclear. Research. (CERN) 


rank a group of accelerators. comprising those. at. Hamburg, Brookhaven 


(which was relegated to this second group when the Fermi Laboratory 


began to_operate) and Argonne. Cambridge, Daresbury, Rutherford, and 


perhaps. Serpukhov were placed by the high-energy physicists interviewed 


ina third. division_of accelerators, the scientific achicvements_of which 


centres in the second group: The relative positions of NINA, and of 


Nimrod, the. proton accelerator at Rutherford Laboratory, were regarded 
as having remained fairly static during the period; neither was viewed as 
having ever risen above this third group during the time they were 


operating. Both NINA and Nimrod: were, however, thought to have 
performed substantially better-than the remaining Russian accelerators; 
which were placed in the fourth division.- _ - 

Over the first half of the ten-year period, during which all four electron 
synchrotrons were operating, and the work of the greatest impact was 
being carried out, the users of Hamburg were-thought-to have. contriouted 
more to scientific progress than those of the Cambridge facility, and 
considerably more than those of NINA; the accelerator at - — 


Taking the -period from 1969 to 1978 as a whole, however, there is 
apparently little or no- difference between the total contribution. to 
scientific knowledge made- by the Cambridge-users (from whom the last 
publication appeared in 1974) and by research workers using the Dares- 
bury accelerate: °- both case:, the contributions seem to have been much 
greater than ° =: * i Yerevan, but significantly less than those of 
Hamburg. 


The Significance and. Limits of Assessments of Past Scientific Performance 
- Since Dr. Alvin Weinberg’ cS ai discussion. of the cviteria tor scientific 
choice. there has been little work carried out to answer such questions as 


“How well is a particular piece of research done?” and “How much impact 
on the advance of scientific knowledge does it have?” It is desirable to try 
to answer these questions in a systematic way. 


In the case of our assessment of high-energy physics; we can n conclude 


~ 
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that the scietititic ahevencaa of the users of the Stanford and Hamburg 


accelerators appear to have been greater than tho: - from the. Cambridge 


and Dares synchrotrons, both. of which_seem | ta. -have made ontri- 


question. It should be borne in mind, however, that the. cambridge 


accelerator was-closed in 1972 and was considerably less expensive_than 


the machine-at Daresbury. Moreover, work carried out on NINA yielded 


fewer experimental papers than its competitors and these were not_on 
average much. drawn. “upon. by other scientists. Nor did NINA produce. a any 


sigbeicie ol sleek as ‘beiiig of high quality. But by the. time this and 
other work at Daresbury was carried out, the interests. of high-energy 
physicists had generally moved on to other areas, so the influence of this 
work was not as great as it might have been a few years earlier. 

‘What, then, is the relevance, if any,-of these results to science. policy? 
When the decision was made to close NINA, fio systematic study of the 
type we have undertaken was carried out. The decision was- -undoubtedly 
made on the basis of some form of assessment by-eminent scientists. Such 
assessments could now be supplemented by studies of the sort we have 
undertaken. 

If. making major advances in scientific knowledge were. the only criterion 
used. in policy-making, then the decision to close down NINA was surely 
the_correct one. Although this is.only one factor to be. considered in 
determining the allocation of scientific resources, it should be one of the 
most_ important comsiderations—especially - -in basic science -where the 


primary objective is to contribute to the advance of human knowledge. 


Nevertheless, there are other issues to be considered; in particular, the 


eration” electron synchrotron, and the. aims of its users were inevitably 


restricted to improving on earlier work and carrying out more detailed 


experiments. _-.. _---_- 
_ In addition, past t performance may give only a limited pctre of future 


influence at the time; would continue to be cited over a long “period 


because of the accuracy_of the measurements. We did not, however, find 


a great deal of support for this view among high-energy physicists; except 
among those who were users of NINA. 
- Some would claim that, even if it was never a-world-leader in the field; 


the Daresbury accelerator nevertheless played a fundamental role in 
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training British. high-energy physicists. _providing them. with. the. oppor- 


tunity to construct and use highly sophisticated equipment. In this respect, 


NINA might | be seen as an “entry cost” to be paid.so that British scientists, 


mainly in. the-north ofthe. country,could join the worid community_of 


high-energy physicists..Many_of the experimentalists from Glasgow, Lan- 


caster. Liverpool, Manchester and Sheffield Universities currently using 


accelerators overseas, part::»-aily at CERN and. Hamburg, gained their 


first experience of high-energy physics on. NINA._Whether or not they 


could have gainec this necessary experience elsewhere, for example at the 


Rutherford Laboratory-or at CERN, is a question we cannot answer. It 


should. be noted, however, that_in_the discussions leading up_to_the 


Original decision to build the Daresbury accelerator, there was little or no 


reference_to the need for NINA_as_a training facility. This_particular 


justification for the accelerator was only invoked at alater-stage. _. 


_The _past. scientific performance. of research groups or centres_ should 


never be_considered_in isolation. Nevertheless, in_making policies which 


to add. to the sum of, scientific knowledge rather than to create educational 


or technological. benefits. past research performance should clearly be 


given. great weight. In this paper, we have endeavoured to show that it is 


now possible to arrive at reasonably systematic assessments of the research 


performance of scientific groups that might be of use in the making of 


science policy. 
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comparative pts piel of pete base reseaj st: facilities (and their 
associated user groups) working in the seme specislty, and appied this method 


of ‘converging partial indicators’ to an evaluation of the conibutions to science 


_-made by 8 number of radio telescopes. In this paper, ve employ this 


methodology to evaluate the scientific performance of various optical telescopes 


— in particuler, the 2.5-metre Iseec Newton Telescope, operated as 8 central 


facility by the Royel Greznwich Observatory in South-East England. For several 
years, this was Britain's cniy-mejor optical telescope, es weil as-being the largest 
_ Such instrument in Europe. We compere its performance over the ta: decade 

-- with-that-of three American telescopes of jz 


This paper has three sims: first. to ascertain whether the method of 

converging partial indicators, originally applied to radio astronomy, provides a 

. more general policy tool that can be extended to other specialties; second, to 
determine just | how successful each optical telescope has been in producing new 
astronomical knowledge over the past decade; and, third, to discuss whether our 
results on the comparative scientific performance of the Isaac Newton Telescope 
may have any implications for Bnitish astronomy policy in general. 


- 


Ben R. Moen 


Astronomy has had a Jong and (in large part) distinguished history 
in Great Britain. Prominent in that history Fave beer the Royal 
Greenwich Observaiory (RGO) and the Royal Observatory, Edin- 
burgh (in recent years or the UK Schmidt and Infra-Red 
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central facilities. or- astronomers in a number of universities. 


tical astronomy was wrested from Britain by scientisi:in thc United 
States, who exploited the large, modern telescopes constructed on 
the basis of generous donations from philanthropic industrialists. 
Immediately after World War II, in an attempt.te_restore British 
opticz: astronomy to its former glory, Scientists put_before the 
British. Government - proposals to build. a magnificent__new 
telescope, to be called the Isaac Newton Telescope (INT), second 


oo 


= for exemple, instrumental. innovation, or the training of young 
astronomers? In what follows, we shall allempt to provide answers 
to some of these questions. .- - —-__- : 


In two other papers,‘ we have explained why there is an urgent 


and increasing need for systematic assessments of research perfor- 
mance in basic science. We pointedtc main reasons. First, ex- 
penditure on certain types of scientifir arch has escalated rapid- 
ly; particularly in the Big. Sci _ thigh-energy. physics, 


adjudication, the method traditionally favoured by science policy- 


makers for establishing priorities, seems to be facing increasing dif- 
ficulties as a tool for determining the distribution of scientific 


CO’: 
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resour. .* The problem is that, in Big Science svecialties, there are 
typica!:y only one or two major experimental facilities in each 
country. These asc generally operated by central staff: who are 


responsible for most construction, technical development and ser- 


vice activities; but who also _use the research facility fora signifi- 


cant proportion of the time, often in conjunction. with external 


users: With research concentrated at such cenires, a large section of 


srned_will have some 


the_nation’s scientists in_the specialty con: 


political interest in all major capital. projects proposed to. the ap- 


implications for- the. system. of. peer-adjudication. ncrmally_used in 


determining. whether_to- support new projects, since the effective 


operation of such procedures depends on. the existence of a. consti- 


tuency of ‘neutral. peers’ able to offer impartial advice. on com- 


peting project proposals, and few, ifany, such ‘disinterested’ peers 


may exist. This problem may be all_ the more- inportant. in-an 


economic slimate where little growth can be expected in the overali 
science budget, since - continued progress.in science wiil. depend 
more aiid more upon the existence of mechanisms capable of ensur- 
ing the rapid shift of resources from old, and no longer. productive, 
areas. of research, to promising. new areas. 7 Consequently, it -is 
essential to begin. exploring. otier methods of obtaining. the 
systematic information needed to complement traditional methods 
of determining scientific priorities. This was the overall objective of 
the. project of which. this study fornis.a part. a 

What are the principal criteria that should be. ‘employed 10 
establish Scientific priorities? Weinberg was one.of the first-to ad- 
dress thi; problem, putting forward a number of-interiial and exier- 
nal criteria for scientific choice.’ However, while he (and other 
writers who contributed to the subseytient debate) went some way 
towards. ‘providing, a ae for. assessing external benefits 
technological spin-off, (Geant, there was ‘considerably less 
ees in establishing criteria to Pe. used. in assessing - internal 


tor the reasons ouilined earlier; it is is becoming essential i -in today's j 


cha nnged ¢ economic. climate that some consideration be § given to the 


question of their systematic application. 


_The. criteria suggested by Weinberg provide an appropriate Star- 


ting peint: '9(1) Is the field ripe for exploitation?; (Il) Arethe scien- 


tists in the field really competent? As we have argued elsewhere, '! 
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the first of these is best separated into two pr: 5 (la). What is the 


state of the field itself. — is it :ipe for exploitation?; {ib) Hew well 


placed are those scientists requesting funding in relation to their 


likely competitors —do they possess some advantage (in time,-or-in 


instrumental capability, for example) enabling them to exploit. the 


field. ahead of such compeiitors? Of these. three criteria, the first 


(a) is probably least. amenable to. systematic asscssment. One-can, 


of course, ask researchers .intha’ scientific field for their opinions; 


but few are likely to admit that their. field is not ‘ripe’ for expioita- 


tion (in the sense that some progress could always be made if they 


were allocated more Lyesetie Seaaee la would indeed. be. better 


deployed Then one can clearly appreciatz c the er treme dif ficulty of 


ver er finding anyone in a position to answer it. adequately. = =| 


Fei iziternal-criteria_ Ib -and II, however, more systematic data 


should be obtainable. For the latter, one of the most relevant picces 


of information concerns the past research performance (the ‘track- 


Agi es os — tat is, the php! of previous contributions 0 


we are not claiming that track-record i ig the sole factor ae 


future research performance ---merely that itis one-of the. most 


significant predictors. Undeniably, there are other factors, such as 


the field's ‘ripeness’: but, if these are So ill-defined that they cannot 


be evaluated in any systematic manner, then there is no alternative 


but to assume that a major. new project stands a greater ¢ chance of 


being. carried out successfully by researchers or institutions with a 


strone *rack-record over recent years, than by those with a poor 


record: Sa eee ees ee 
- itis with the aim. n of assessing the scientific irertorinaiice: of cen- 


tral fa acccities: and their_user_ salty th relative to” that of their com- 


— — that i is, variables determined. sd gacily the ae of the} pai. 


ticular contributions, and partly by ‘other factors’. For example, 


one of the possib’e ‘ndicators (the number of scientific papers pro- 


duced by the usc: : of each facility) reflects not only their output of 
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scientific ideas, but also the publication practices of the researchers 
concerned, and, in particular, the emphasis. placed on publications 
in determining. access to furids and promotion prospects. Some of 
the partial indicators refer to the overall contribution to scientific 
progress made- by the users of each facility (for. example: the 
numbers of publications,-citations, and highly cited-papers; peer- 
evaluation rankin,*, while others take account-of differing scales 
of rese2"ch activity and | reflect their | “productivity: (or output per 


cher; the cost per. publication: and s so aii), If these partial indicators 
are to yield reliable results on comparative research performance, 
then the influence of the ‘other factors’ must be niinimized: the test 
that this condition has been met is convergence between. the in- 
dicators. On the basis of other assessments of Big Science facilities 


significant. results can be obtained provided the. following precau- 


tions are observed: (1) the indicators are applied to user groups, 


rather _than_to individual scientists; (2) a range of indicators must 


be employed, each focussing on different aspests of a central facili- 


ty’s performance; _(3) the indicators can only be meaningfully ap- 


plied to matched _facilities and_user groups, comparing ‘like’ with 


‘like’ as far as_possible; (4) the indicators based on citations must 


be seen as reflecting the impact, rather than the quality or impor- 
tance,” of the research work; (5) because of the imperfect or par- 
tial nature of the indicators, only in- those. cases where they yield 
convergent results cari it be assumed that the influence of the ‘other 
factors’ has been kept small (that_is, that the matching of the 
facilities and user groups has.been largely successful), and that the 
indicators provide a reasonably reliable estimate of the contribu- 
tions to scientific knowledge mzc> using the various research 
facilities. 4 

-Using_this method, we were able to identify with a certain degree 
of confidence sic world leaders, or ‘first division’ telescopes, 
aed radio- astronomy, - whose research performance has been 
samen — = which i in turn, | have been appreciably more successful 
than certain ‘third division’ telescopes. '* We Argued that, if this 
procedure was carried out. for other- basic science Specialties, this 
would provide valuable information for pcticy-makers responsible 
for the distribution of-resources berwecn specialties. (If other fac- 
tors were equal or indeterminate, they would normally prefcr to 
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support a first-division facility in one basic science specialty rather 
than a second-division facility in another.) What was not.clear 
from our earlier work, -however, was the extent to which the 
methodology constructed for the evaluation of radio telescopes 


specialties, particularly when those facilities were not-amongst the 
world leaders for their field (as two of the four in the radio 
astronomy study proved to be). Hence, it is partly to assuage such 


doubts that we present below an assessment of the scientific perfor- 


mance of. - the large 98-inch. (2.5 metre). Isaac Newton Telescope: at 


instruments operated in- other countries. Although Jess. expensive 


than high-energy. physics, optical astronomy nevertheless absorbs 


considerable- financial and scientific_resources.. 4% Within Britain, 


between 1967 and 1979, the main capital facility was the INT, 


although by the mid-1970s the country also had a half share in the 


3.9-metre Anglo-Australian Telescope in the Southern Hemisphere, 


and is currently engaged in building a 4.2-metre tclescope.on La 


Palma in the amy Islands. As astronomers themselves admit: 


Most of the telescopes that we use are very expensive and its our responsibilty 
to ase them effectively. It seems reasonable from time to time to take a hard iook 
at the productivity of those telescopes and te ask whether we are getting our 


money’s worth in our use of them.!? 


Hence, in. addition to testing the method of converging partial i in- 


dicators on another specialty, the other main aim of this paper isto 


. "amine the scientific track-record of one such expensive facility 


and its user group,- comparing it with that_of various competitors. 


We should then be ina better position to reflect on the relative suc- 


cess of British post-war policy for optical astronomy. First, 


however, a short digression on the history and background fo the 
Isaac Newton Telescope is necessary. 


and the Isaac New’ on Teese, 
The INT has had OR. * st chequered history: 8 Originally c9 con- 


ceived in 1946 as the project that would restore Britain to its former 
pre-eminent position in: optical astronony, it ok 12 years from the 
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a total of 21 years to finish the project, although even then | it was 


not available for use. by universitw astronomers until the end of the 
following year. While we need not. be concerned here with the 
reasons for the delays in- building the telescope,” several of "the 


more important decisions involved should be mentioned: Two were 


clearly taken:i -in, or around, 1946. First, the astronomers concerned 


more quickly. Second, it was decided that the INT _should be 


operated as a central facility by the Royal Greenwich Observatory, 
rather than by. a_new ‘Central. University _Observatory’_— __as 


Plaskett, . Professor of pelea een gage and the 


Although the : aibilij ot a an Overseas site was only acm con 


sidered, in open and public discussion, from 1955.on, our evidence 


(see later) suggests that the decisior: to build the telescope in Britain 


was effectively taken much earlier, and that some astronomers ac- 


tvcly supported such a-_move. Despite tie evident unsuitability of 


the British climate_for astronomical observations,” senior 


astronomers argued that the telescope should be sited at the RGO 
rather than overseas, on the grounds that only then could the close 
links. between theoretical and observational astronomy — links 
which are essential to the health of both these fields, and of the 
universities — be maintained. - 

In the. flood of enthusiasm for the. exciting new project, sid 
perhaps because of the ease with which government backing was 
obtained, astronomers made one crucial oversight. In-the thirteen 
years between being given the finaiicial go-ahead. and the awarding 
of the construction coiitract in 1959, no systematic site-testing was 
carried. out at the proposed-location for the telescope. Techniques 
of site-testing were well-established in. the United States, and-siich a 


. procedure was regarded there as absolutely essential in selecting the 


best available site for any major new telescope.” It would have re- 
Quired neither a great deal of time, nor of money, to carry out 
similar tests in Britain to establish just how suitable the proposed 
Jocation would prov:. 
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The INT. was officially i inaugur sted i in 1967. Within a few years, 
the unsuitability of the site was fuliy i avealed, ond in 1975 the deci- 
sion was made to move the telescope to <a Faima, Fowever, just as 
the costs of the original telescope roe" fran ae 4 instin’ csamate of 
£200,000 in-July 1946 toa final cor 6) + +e ately Fi-million, :o 
the costs of moving and upgrading # 32° a ‘trom £1.76 
million in 1975 to the best part of £7.5 ee 1 GA, © van far 
greater than the original cost of the telescope, een ac aitowing 
for inflation. The telescope has recently been waved. :9 :x,¥ would 


seem to. be an. -appropriate_ time to ASSESS. its aca 2 record. To 


used at the time to justify the sizeable Silene “tt in 1957, 


Sir Richard. Woolley, Astronomer Royal and Director of. the RCO, 


predicted thatthe INT would ‘do much to prosnote a vigorous 


resurgence in the practical as well as the theoretical sroblems. of 


astronomy in this country’. * Has the INT been abie to fa! fil this 
promise? And, if not, what. went wrong?_- 

Our results are perhaps also of some relevance to another, wider 
set of policy. issues relating to the British plans for a. Northern 
Hemispiere Observatory (NHO) on La Palma’s —_and in par- 
ticular the decision that the RGO should be. responsible forthe con- 
struction and operation of the new. facilities. These stem from the 
fact that there has in recent years been some criticism concerning 
the overall role of the Royal Observatories in British astronomy. * 
in interviews. with us, critics argued that the present institutional 
structure of British astronomy (with the Royal Observatories essen- 
tially being part of the same public dody.— the Science and 
Engineering -Research Council — that controls-the. funding of 
astronomy) is not ideally ~iited for ensuring a flexible and. suc- 
cessful-_natioral policy to.,ards optical astronomy. They claimed 
that this has. led in. the Past to an overconcentration Of resources on 
universities planning and. operating their own-central facilities (as 
they do at the American Kitt Peak National Observatory).””_ - 

_In view of this latter criticism, it-is clearly desirable to have 
systematic information on the research record of the INT.-Success_ 
or failure in the research record of a large centralized facility will 


depend partly on the users (both internal and external); and partly 


on the effectiveness with which the organization. 1 ponsible | for its 
construction, instrumentation, and maintenance (in this case, the 


RGO) carried out its duties: 
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: -Somec ‘consider ation-was given. to this latter-i issue in the sii 
1970s. _ The -Stience_Research Council established 2 Northern 
Hemisphere Review Committee to plan the future of British optical 
astronomy in the Northern Hemisphcre. Although the Committee 
was not unanimous in its conclusions, a majority” recommended 
that it-would be in the best interests of British as:ronomy that the 


NHO be run, not Sy tke RGO, but as a new ‘third centre’, indepen- 


dent of the two Royal Observatories, and. giving. university 


astronomers a oes greater. r possibility. of | Participating in Fi stages 


ecounmendatna | however, the members of the Committee did not 


have the systematic data (for example, on publications and cita- 


tions) to justify, on scientific grounds, why.a new centre should be 


given responsibility for the NHO. In part, this was because the INT 


had_only. _been_operating for a few. years, and, although. certain 


astronomers were already dissatisfied, this was rather early.to make 
a formal study of how well the INT had operated under RGO cor.- 
trol. Nevertheless, 2 majority of the Committee did feel that, while 
the RGO was capable of coping with the needs of more traditional 
astronomy, it was not so well placed to meet the demands 
associated with operating a large modern telescope.In addition, the 
Committee produced certain financial figures which suggested that 
the RGO, when compared with American observatories, was 
relatively expensive to operate. However, in the-absence of data on 
the ressarch performance of the INT, the enstiing debate over tlie 
Cummittee’s Teport, appeared, to outside observers at least, to be 
conducted more in political than scientific terms,” and eventually 
the Science-Research Council decided. not to implement 1 the Com- 
mittee’s main recommendations. *In the final section of this paper, 
we discuss how the scientific assessment results presented-here, had 
they. been available atthe time, might have affected the decision in 
1974 on the organization. of the NHO.— a decision involving con- 
Siderable resources, arid affecting the whole future of Eritish 
astronomy. 


The INT and its Closest Corapetitors ; 


In. applying the. inethod of converging. partial indicators to i ESSCSS 


the. cana a telescope like the INT, the first ee 19 


Ol 
hh | 
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ingful comparisons can be made: Whereas this was not too diff fi cult 


in our earlier study of radio astronomy?" —.a specialty where there 


are just a few major facilities, each with a fairly- stable. research 


group concentrating its work almost solely. on that. facility. — op- 


tical- astronomy is characterized by a rather larger number_of 
facilities, each drawing its users from a much wider scientific com- 
munity. Indeed, some optical astronomers - move between. in- 
struments frequently, seeking observing time at anumber of obser- 
vatories;-the most eminent astronomers typically gain access. to 
perhaps-four or five major telescopes each year. Although a few 
major observatories are still operated by individual universities (or 
small -Broups of universities), most.are run as national centres — 


the commiiient of i many researchers to more than one telescope, 


one can only hope to evaluate the output from a t¢:..scopc (or set of 
telescopes), rather than-that of a-fixed resear-® eroup. So which 
telescopes are most Similar tothe INT? | 

_The main determinant of the researc. :*- alia of optical 
telescopes is the aperture (or, more stri*. +. :*: Jight-gathering 
power). Hence it would be somewhat ws.:~t%i+":" to compare the 
scientific performance of the 2.5-metre i!?2 «ith that of the 
5-metre telescope on Mount Palomar, because of the significant 


difference in size. We have therefore chosen to- -compare the INT 


with the.3-metre Lick telescope on Mount Hamilton-in California, 


the 2.1-metre. es ai the Kitt Peak National Oiservatory 


Tololo Inter-Amer: can 1 Observatory (CTIO) in Chile (which i is run 


by the same group. of American universities that operates the Kitt 


Peak. Observatory). *? The latter_two are, like the INT, national 


facilities. The Cerro Tololo telescope is rather smaller than the 
INT, but. -by including i itin the study we should be able to gain some 


idea of the potential results that_Britain might have obtained if it 


had elected i in 1946 to. build a telescope somewhat smaller than the 


also serves astronomers. from a. number of. University of. California 


campuses, and, therefore, in terms_of the institutional and 


geographical dispersion of its user community, is not too different 


iA nature from the INT. All four telescopes. are_of fairly conven- 


tional aesign: all- were e operating. throughout_ the. (10-year, period 
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One iminediave complication in comparisons ae oe ese 


telescopes arises from the fact that they form part of « Po 18s 
which-operate several tciesco¢es, and carry oul 2 var... -. tasks 


apart from astrophysics. The problem is that for certain types of i in- 
formation (like annual operating costs), although : ic is simple to ac- 
Guire data for the observatory as 4 whole, it is extremely dif ficul: 0 


_ disaggregate these -for individual telescopes. In. Table }, we 
’ therefore list not only te main details of the four telesecpes under 


comparison; but also those for the other telescopes at each of the 


. Observatories. 


Input Measures 


Before considering the outputs from these telescopes, and asses sing 


various inputs hat have been used in ‘creating those ‘pulping. The 
aim here is not to derive precise figures on the costs.of carrying out 
astronomical research; faced our point is thal, if there are ssignlti- 


these clearly have to be “taken i into account in evaluating ihe out. 


puts.- -— _.. __. 


__One obvious input is the overall cost_of running - the telescope 


ferent accounting. procedures (tor example, as te what counts as 


‘telescops_operations’ and what as ‘central facilities’); and bec. .e 


‘each operates several diffeient research facilities, it is extremely dif- 


ficult to obtain comparable figures for the four telescopes. We have 
therefore adopted the following procedure,. which, although not 


observatories. _ ne 
Beginning with the fi igures for the total annual ¢ operating costs of 


each vacchdgy adel we e have. first suvtracted from this an appropriate 


allow for their solar, planetary and space work, a and another. twenty 


percent for various central facilities used by astronomers to. process 


observations obtained on telescopes other than those at Kitt Peak: 


In the case of tite RGO, thirty percent of the total observatory costs 


was deducted for the work on positional astronomy and providixg 
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: TABLE i- 
Maia Stellar Telescopes Operated by the Gino Tototo. Kitt Peak; Lick; 


and Royal Greenwich Observatories, 1978 


Cenire Telescope Completion Approximate 


date -eapital-cost- 
(for post-1945 
telescopes)* 
oe 
Cerro a&metre 1974 
Tololo. _- 1.S-metre- 1568 
Inter-American Yale 1-metre 1974 
Observatory 0.9-metre-- - -- 1967 
(CTIO) Michigan ‘Cans Sauna = ee 
- 0.6/0.9-meire 1967" 
Lowell 0.6-metre 1969° 
0.4-metre 1965® 
0.4.metre 1961" 
Kiit- a-metre 1973 $11.10M 
Peak 2.1-metre 1963 $ 2.90M 
National - 1,3-metre 1966 $ 058M 
Observatory 0.9-metre 1966 5 7.3M 
(KPNO)* 0.9-metre 1960 ad 
0.9-metre Coudé feed 1972 $ 0.57M 
0.4-metre 1963 a. ae 
GA-amesre 1962 ae 
Lick 3-metre - 1959 : 380M 
Observatory 0.9-meire - - - 1895S ~ 
C.9-metre refractor 1888 cee: 
©.8-rietre 1964 % 0.58M 
0.5-metre arirogregh 1939 _ 
0.3-metre refractor 1882 = 
RGO 2.5.meire 1967 £ 0.96M 
0.9-metre 1934 = 
0:8-metre ___ 1890s _ 
0.7-metre refractor 1890s _ 
0:7-metre refractor 1894 _ 
0.3-metre refractor 1859 ~ 


a. No attempt has been made to adjus: those costs for inflation: 


>. This-is the date the telescope was-moved tc-Cerro Tololo. 
. Kitt Peak also operates several solar telescope facilities: 


m oie 
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i : ‘TABLE? _ : 
Approun:: ; _ Annwal 1 Operating Casis* and Numbers of 


«.stronomers Using each Telescape 


CTIO) = KPNO_—sLckk RCO 
Total observatory 1970 12.684 £0.37M_—£0.70M* 
operating costs 1974 £3:33M £0:57M  £1:35M> 
1978 ~ £2.1M £46.77M ‘“-£0.90M  £2.00M‘ 
Estimated cosis- 1970 £1.34M¢ £0.37M —£0.28M 
Of steliar astrophysics 1974 £1:67M £0:57M = £0.54M 


work at the observatory 1978 ~ £2.1M £2.39M  £0.90M = £0.80M 


Percentage of the stellar 

astrophysics costs 

apportioned:t tothe - 9.8% 17.2% 82% 12% 
ticular telescope being 


Estimateo annual 1970 $023M® 30M —-£0.20M 
cost of the telescope i 0.29M —EU.47M £0.39 
being assessed 1978 ~£0.7: : :0.41M £0.74M = £0.58M 
Capital costs: 1970 £0.02M £6.92M —£0.05M 
amortized over the- 1974 £0.02M = £0.02M_—£0.05M 
life of the telescope 1978 £0:03M  £0.03M £0.05M 
Total annual costs 1970 £0254 £0:32M —£0.25M 
1974 £0.31M £0.49M £0.44M 
1978 £0.44M £0. ia £0.63M 
Appropriate number of 1979 40 60 43 45 
users* 1974 65 6s 45 100 
(978 8 90 47 7 


_a. These figures are approximate estimates only, obtained-using the procedure 


described i in the text. To convert doflars | ints pounds sterting, the exchange rates us- 


ed. are those nee in the National Institure Economic Review, Vol. 86 (1978), 
Table 25. - e 


b: These # are.the annual co.ts s afier subtracting the cats ¢ of the Radcliffe and Cane 
Observatories then operated by the RGO. 


_& The cverall RGO Sadget in this-financial. year «0% £3 45: ‘miilian: but this in- 
cludes a large riunpower and capital-investment-in the cew NHO facilities which is 


difficuts to SEDATEIE Cat from oral costs: Using the figure for the costs in 1974 (the 


last year in which the NHO expenditure was kept separate from that of the RGO), 


erie? 
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TABLE 2 (continued) 
anda assuming a 0% increase i in- ag costs in thei intervening 45 years rs (the Reiail 


Price index in:fact ros by 80% over this period; so this calculation probably under- 


estimates the-RGO costs), leads to a figure of £2.0 miition for the estimated costs of 
the RGO in 1978. : 

-~d. The figure | for the: total costs of Kitt Peak i in i970 includes a a iarge capital i invest- 
ment in the new 4.metre telescope, Since the Capital expenditure in that year-was 
almost -certainly greater than the ronning costs of: this telescope once it began 
operating, use of Abt’s papidinssea almost certainly results in an Over-estimation of the 


Overseas. The relevant fig igures are given i in the second row Sat able 
2. It should be noted. that these costs include. ee on.those 


operatic a jaige modern iaeesipe. titae ned sage i is to apportion 


the total costs of astronomical work at each observatory between 


all the telescopes operated by it. In doing this, we have used the fin- 
ding by Abt that, within KPNO, the annual-operating costs. of 


telescopes vary as the 2.1-power of the aperture. Assumiig that 
the same relationship holds spproximately_for telescopes. within 


each of the three other observatories, we have calculated the 


percentage of the total astronomical costs that must be apportioned 


to_the four telescopes. under consideration here: As-can be seen 


from Table 2, the figure varies from about ten peri?nt for the 1.5- 


metre telescope at Cerro Tololo to 82 percent for the Lick 3-metre 


telescope. Using this percentage figure, we can heii estimate the 


approximate total costs of supporting and carrying out research 
with the four telesc:spes. 39 

If Abt’s 2.1-power cost/aperture relationship were ae to hold 
between observatories, we should expect that the operating costs of 
the. CTIO. 1.5-metre telescope would be about half those of the 


KPNO 2.1-metre telescope, the INT to be 45 percent more expen- 


sive, and the Lick 3-metre telescope to be just over twice as expen- 
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nature of ‘the estimates in. Table 2. one can see that the costs are in- 


deed roughly intheseratios. - © - . |. : : ..:.: 
_ In addition to annual operating costs, there are also the emital 
costs to be taken into consideration. Normally; big telescopes can 
be expected to have a long working life — perhaps fifty vears or 
more. * if capital costs are written off over a period of fifty years, 


this adds only approximately £0.02 mill:on*’ to the annual running 


costs of each of the American telescopes, making little difference to 


the figures quoted in Table 2. However, in the case of the INT, 


because this was only used by university.astronomers for about 10 


years (and by RGO staff for 12 years) before being refurbished and 


moved.to La Palma, at.a cost.not greatly dissimilar from_that.of a 


new telescope. (see_note 23), its capital costs must. be written. off 


over_a rather shorter period. We have used a figure of twenty. years, 


on_the grounds that the telescope was.used in Britain for some ten 


years, and on the optimistic assumption that half of the telescope’s 
components will be re-used on the new site. This adds £0.05 million 
to the annual costs of the INT. However, as the figures for the total 
costs on the bottom row of Table 2 indicate, this makes little dif- 
ference to the ratios- of costs for the four telescopes. Since- (as was 
seen above) these ratios are roughly what one would expect on the 
basis of the relative sizes of the telescopes, one must conclude that 
in world terms the INT was nor significantly less well. supported 
than other similar telescopes. Any difference between its scientific 
performance and that of the American telescopes cannot therefore 
be-attributed-to a comparatively low level of funding. us 

In our studies of radio ) astronomy i and high-energy. physics, a se- 
research at each facility. : However, as was mentioned earlier, it is 
impossible to identify a stable ‘user community’ for each of the op- 
tical telescopes. Ini the botiom row of Table 2, we give thie approx- 
imate number of astronomers using the telescopes in particular 
years, but it must-be appreciated that some.of these astronomers 
were permitted-only four or five nights of observing, while others 
had twenty or thirty. The former is more likely to be the case.at the 
national observatories (that is, at CTIO,_KPNO, and the -RGO), 
while the latter applies to many of the University of California 
astronomers using the Lick telescope. For them, perhaps the ma- 
jority of their research will be carried out on the Lick telescope, 
while the Kitt Peak 2.1-metre and Cerro Tololo 1.5-metre 
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telescopes might provide their users with a much smaller fraction of 


their total observing time in any one year: This would then. explain 


why -each_of- the three. national-facility_ telescopes. supports a 


Significantly larger user community than that for. the Lick 


telescope. Because of this_widely differing pattern in the usage of 


the four telescopes, this particular input indicator will not be used 


in any of the comparisons that follow: 


The Assessment of Contributions 1 


Scientific Knowledge 


In 1 comparing tf the scientific o output. from research facilities, perhaps 


the most important data source is the scientific publications pro- 


publishing papers . in. scientifi it. journals ‘that researchers com- 


municate new ideas. and results to other scientists. Although there 


are_undoubtedly other, less formal channels -for disseminating 
research. findings, it can generally be assumed that information 


passing through these channels eventually ends ‘up: in the scientific c 


element in the reward structure under which scientists operate. "9 In 
our work, we have angie taken mies publications as.con- 


which we are concerned ‘compare in their output of scientific 
papers? 


observational results. obtained with the telescopes. It can be seen 
that the Lick and Kitt Peak telescopes have both produced an 


average of ue over forty 5 papers. a | Year during | the: fast decade, and 


fig figure for the INT is seven. An even larger difference i is shown i in 


the figures on costs per paper; research publications from the Kitt 


Peak telescope have cost only just over one tenth those from the 


INT. —. __ a 
_ However, as we. e have been. at great pains t to. > point out in. our 


previous _work, ” publication rates are but a partial indicator of the 


contributions made to scientific knowledge. It is possible that a 


wi 
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TABLE 3 3 


cio KPNO - Lick _ ANT 


1.5-metre —.2. J-metre 3-metre 2.5-metre 


No: of publications in®:- - a 
1969 13 22 33 


j 

1970 - 15 29 29 0 

1971 a 33 35 6 

1972 3s a3 30 S$ 

1973 39 38 «@ 14 

1974 31 $4 37 6 

1975 «& $1 “4 9 

1976 4s 56 51 10 

1977 “4 43 62 10 

1978 39 65 61 - 10 

Aggregate for 1969-78 354 434 422 71 

Yearly average for 1969-78 35 43 42 7 

Approximate cost ---- & gare . 

per paper: 1970 ? Em £10k r68k 
1974 2 £6k £iik £53k 
1978 £6k £7k £13k £63k 


aS = 


ing -all the papers in-randomly selected volumes of astronomical journals — 
that the publication lists obtained inthis way are-between-85% and 95% complete. 


Hence the figures in Table 3 should be acccrate to within 10 or 15%; which is suffi 
Gent for our purposes here. ----- -. -. - 


- b. Because the number-of papers published in any one year is relatively small, 


there are Significant. random-fluctuations from year to year. In an attempt to 


“smooth out’ these fluctuations, we have taken an average of the. numbers published 
Over.a three-year period, beginning with the-year in question. Thus the 1970-figures 


ase_calculated by averaging the numbers of pablications.in 1970,.7)_and 72, and 


dividing by the total-costs of the telescope in 1970. Similarly the 1974 figures are bus: 


ed’on the average number of publications between 1974 and 1976. (For the 1978 


figures, however, we have used the numbers of papers published in 1978 only, since 
the 1979-and 1980 figures were not known at the time of completing the work 
teporied here.) This procedure also allows for the delay (typically one year) between 


carrying out observations and publication of the resulis: 
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_ Ss se = TABLE 4- - She ki eet 
Ouipal fadicators for Optical Telescopes: Clations 


cTIO) = -KPNO - Lck _ INT. 
1.S-metre Zi-metre —3-metre 


f 
; 


=a 


Total numberof 1970-30 #0 210 
citations to all 1972 150 290 340 
observational - 1974 140 
papers published 1976 


340 
19 6 
since 1969 1978 880 1200 1460 
= , 


8 
= 
Saas: 88ia% 


Number of —- 1972 150 440 
Gtations to work =—-1974 300 360 490 
published in the 1976 460 $60 590 
preceding 4 1978 $80 

years onty 


= 
=) 
8 
= 


Average number —-:1972 1.7 2.2 
of citations per 1974 ] 2.3 
paper for work 1976 2.6 2.8 
published in. - --- 1978 3.3 33 
preceding 4 years 


OVERALL AVERAGE 2.5 2.7 3.5 


- 

. -) 

~ 

- 
ww. 
|| a A 
vip Nie 
aoe. 


kg 
oe 


Number of n= 12 21 3 4t 
papers gaining n nals 16 20 23 
Of more citations «=n = 20 4 $ 
in one year n=30 . 1 


 & imo By la | 


Number of-times n= 12 
highly cited n=Is 
papers received n=20 
oF more. n= 30 


citations in a 


year 


ou Big 
3s 

as 

Cw ans) 


a: This figure is based on a relatively small number of papers (12, compared with 
about 100 for the-three other telescopes), and the high vatue here is largely due to 
one very highly cited paper. 
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telescope with arelatively smail public ication output might ae con- 


tributed significantly to the advance of knowledge if those publica- 


tions have had a comparatively large impact on the scientific com- 


munity. In this respect,.we may regard the number of citations 


gained by all the papers from a telescope as providing a partial in- 


dicator of. the total impact of that telescope’s work. The figurss for 


this have been obtained using <he-Science Citation Index; and are 


given in the first row of Table 4. They show that Lick papers have 


received between seven and- ten times as many Citations as those 


from the INT, the Kitt Peak papers between six _and_ten times as 


many, arid the yas Tololo t Papers between t three and seven. times 


publications. has "beea considerably less than that of ¢ even the 
smallest of the three American telescopes:  9§ «5 


- It: might-be argued that the INT did. not really begin 10 o contribute 


significantly to scientific knowledge until a year or so after the start 


of-the ten-year period considered here; Although the telescope was 


. Officially inaugurated in 1967, it was not. fully_ready for use by 


university astronomers_wntil some time Jater,.and in fact produced 


only one paper in 1969 and 1970. One can, however, examine 


whether there have been appreciable changes_over time-in the im- 


pact of each telescope’s work by looking at the citations to only-the 


most recent work gin this case, work published during the preceding 


four years). Tie figures im the second row of Table 4 do reveal a 


significant increase in the impact of recent INT work — by a factor 


of nearly three between 1972 and_1978 — but there are roughly 


similar increases for-all the three other telescopes, “'so that the INT 


total for 1978 ‘remains fess than a ‘quarter of the. Cerro Tololo 


figure, less than one fifth the Kitt Peak figure, and less than one 


sixth the Lick. figure. - 


_ What factors. gave rise to these apparently. very jntge differences 


in impact between the British and the American telescopes?” Un- 


doubtedly the main factor was the-substantially lower publication 


rate of the former. The figures on the average number of citations 


per paper earned each year by recent publications from the four 
telescopes suggest that, on average, INT publications have had a 
very similar impact to those from the Kitt Peak telescope, and pro- 


bably slightly. more impact than papers from. the-Cerro Tololo 


telescope. Lick papers, as $ might, be expected. given the g greater.s size 


astronomers using the INT had published as many papers as their 
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colleagues ttsing the American telescopes, then; provided that there 
was | no. ‘dilution’ ‘in the impact of those papers, the. overall impact 


similar to that on the three other telescopes. - 
_ There is one further indicator based on citations that needs t to be 
considered-— namely, the number of highly-cited papers produced 
by each telescope. While the total number of citations, together 
with average-citatiotis per paper, may give a geficral indication of 
the impact of work from a particular telescope, it may-not reveal 
whether-that telescope has been responsible for any of the occa- 
sional-‘discoveries’ that play a major role in the advance of science. 
Scientific progress can come about through both incremental and 
revolutionary change;- and it is conceivable that a facility having 
relatively low publication and citation rates might still be judged to 
have -had a -substantial impact on. the. advance of scientific 
knowledge if it could claim a few-major discoveries. The distribu- 
tion of such discoveries. between-the four telescopes assessed_here, 
while it may be masked in the figures on total citations and cita- 
tions per paper, should show | up in data on the numbers of highly- 
cited papers.“ ° 

_For these four telescopes, the top two. percent friost-cited papers 
gained t twenty or more Citations in one year. As can be seen-from 


the pega row of Table : 4, the. Lick telescope ; produced thir- 


Patna is made for some influential | papers bene fe 


than cithers (bottom row of Table 4), the differences between the 


four telescopes are no jess pronounced. 


_ Finally, how-do the results obtained using t the various piublica- 


tion and citation indicators compare with those from peer- 


evaluation? To answer this, we asked about fifty astronomers (in- 


cluding a large fraction of the research staff of the Kitt Peak, Lick 


and Royal . Greenwich. Observatories, and some_of their outside 


four telescopes considered. here. We found, however, that many 


astronomers experienced difficulty_in_distinguishing work carried 


out on these telescopes from that performed on other telescopes at 


the same observatories. Consequently, the. peer-evaluation results 


that follow refer to the overall output of all the main stellar 
telescopes at each observatory (those listed in Table 1),“ although 
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somie important. non-quantitative data were obtained on the record 


of the individual telescopes with which.we are primarily concerned. 


_AStronomers were. invited to identify the main scientific con- 


tributions made. by. telescopes at their own observatory (or the one 


they use for the bulk. of their research). arid-by those at eleven other 


observatories (including most of the world’s major. optical obser- 
vatories) over the ten years between 1969 and 1978. They-were then 
asked to place these-in rank order (or, failing that, in ‘first-class’, 
‘second-class’, ‘third-class’ and ‘fourth-class’ categories) according 
to the relative magnitude of their scientific contributions. Because 
the telescopes at twelve observatories were involved in this exercise, 
the rankings are on.a scale between 1 (top) and 12 (bottom). The 


average rankings obta:.ied from each group of astronomers are 


shown in Table 5. ae 
_AS in our other surveys of radio astronomers and. high-energy 


physicists, there was-a high degree of consistency in the rankings 


odtained, both within each group of astronomers and between 


them. There were, however, small systematic differences between 


how astronomers ranked the output of the telescopes on which they 
worked (self-evaluation), and how their colleagues elsewhere rank- 


ed it (peer-evaluation). “Consequently, the average. peer-evaluation 


Hea obtained after excluding self- rankings (these. are shown i in 


more ‘reliable indicator. These Show that observational work at Kitt 
Peak was rated 2.6 (second only to the Hale Observatories), at Lick 
Observatory 4.3; at Cerro Tololo 4.6; and at the Royal Greenwich 


Observatory 11.0. This suggests that optical astronomers. judge the 


Scientific contributions made using RGO facilities over the decade 
to have been vety appreciably less than those made using the three 


American observatories. Be eeeeetet eases 
Because these peer-evaluation data refer to all the. main stellar 
telescopes i at each observatory, rather than to individual. telescopes, 


comparisons with results from other partial indicators must.be ap- 


proached with some caution. Publication and citation data all sug- 


gest that the Lick 3-metre telescope has contributed significantly 
more to scientific knowledge than the Kitt. Peak 2:-1-metre 


telescope, while the peer-evaluation. results. putt the Kitt Peak work 


overall. -ahead of the Lick Observatory. The Principal reason 1 for 


Kitt Peak has been the 4-metre telescope, and it is this telescope, 


rather than the 2.1-metre instrument, that from then on has had the 
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greatest influence i in determining. the overall scientific ic contribution 


of the observatory. In addition, there are-several other smaller, bui 


relatively productive telescopes at Kitt -Peak,--particularlv ihe 
1.2-metre instrument. If we had included the publications and cita- 
tions. stemming from observations on all these telescopes, the 
resulting Kitt Peak. figures. would. have been considerably larger 
than those for Lick (even after adding-in the work from the smaller 
Lick reeseopes) “ * Similar considerations apply to. Cerro- Tololo, 
1975. In this case, however, approximate calculations suggest that 
the figures on publications and citations would not have been very 
different from those for the Lick Observatory, in line with the very 


similar rankings that-the two-observatories were given in the Peer- 
evaluation exercise. Finally, in the case of the optical facilities at 


the RGO, the INT has been the principal telescope throughout the 
decade. If papers from the smaller stellar telescopes at the RGO 


were included, this would probably add only about fifty percent to 


the publication and citation totals — leaving them_a factor of six or 


‘more below the equivalent American totals. Overall, therefore, we 


see that, once consideration is given to the other telescopes at each 


of the four observatories, there is a reasonable degree of con- 


vergence between t the results obtained with publication and citation 


_Before examining what conclusions c can be drawn from this con- 


vergence, let us look in. ‘more. detail at what the peer-evaluation 


results indicate. The results in-the final column of Table-5 are 


shown pictorially in Figure 1, which illustrates more graphically the 


differences between the contributions from_the telese2pes at each 


observatory. What the figure and the table do not show, however, 


are changes.in telative positions over time: In addition to ranking 


overall contributions to science. over the decade, astronomers were 
asked to identify changes i in relative positions over this period. It 
would seem from their responses that, at the start of the decade, the 
Hale and Lick telescopes were clearly perceived as the two world 
leaders in optical. astronomy. However, once. the new 4-metre 
telescopes at the-Kitt Peak and Anglo-Australian Observatories 
came into. ‘Operation in the mid-1970s, _ these two appear to have 
eaicacnert described the Hale, Kitt Peak, ‘atid Anale: Alstralian 
facilities as making up the ‘first-division’ in optical astronomy. By 
1978, Lick seems to have moved down to the ‘second-division’ 
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pai eset ee ht __FIGURE! See: toes - 
Average Rankings of Work Carried Out on the Telescopes at 
12 Optical Astronomy Observatories for 1969-78 
(exclading self-rankings)* 


given by the different groups of astronomers. 
For explanation, see text. 


a. The error-bars indicate the root-mican scraare variations between the rankings 
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because its 3-metre telescope found itself at an increasing disadvan- 
tageto the new 4-metre telescopes at observatories -boasting 
significantly better_observing conditions. Besides Lick. other 
members of the ‘second-division’ cited by astronomers were CTIO 
(perhaps now threatening to overtake Lick) and two university 
observatories not included in the peer-evaluation results — the 
University of Arizona Steward Observatory, and the University of 
Hawaii Observatory (whose position was also seen as improving 
rapidly). The users of these were judged to have contributed 
significantly more to scientific knowledge than those of a number 


of ‘third-division’ facilities, including the Texas McDonald Obser- 
vatory, the European Southern Observatory (where the new_3.6- 


metre telescope has so far failed to contribute as much as the other 
4-metre_telescopes.constructod at about the same time), Mount 
Stromlo. Observatory in Australia, and the South. African 
Astronomical.Observatory (whose main telescope was formerly at 


Isaac Newton Telescope and the Russian 6-metre telescope. ___- 

--In. the particular case of work carried _out_on the INT, 
astronomers considered that its impact on. the advance of 
knowledge over the ten years up to. 1978 had been minimal, as comi- 
pared with that of the major American telescopes, and many (in- 
cluding some atthe RGO) had difficulty in pointing to any specific 
scientific contributions of importance. This cannot, in our view, be 
attributed to-ignorance on the part_-of American astronomers of 
research work in Britain, since several were able to identify. con- 
tributions made by the much smailer, 0.9-metre telescope at Cam- 
bridge University, Nor can_it be blamed on chauvinism, since 
American astronomers. rated_the recent_-work-of the-Anglo- 
Australian Telescope quite highly. It therefore seems difficult to 
escape the conclusion that astronomers see the INT as having made 


a relatively small contribution to science over the ten years between 
1969 and 1978.9 


Discussion and Conclusions 


One aim of this paper was to determine whether the method of con- 


verging partial indicators could be applied beyond the specialty of 
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tadio astronomy, whee it was fin rst used. Despite various dif: 
ficulties in. making direct comparisons, the partial indicators 


presented in this paper have yielded generally consistent results. 
The method appears to be capable of comparing the relative scien- 
tific contributions of optical telescopes, and of producing -signifi- 


cant results, With the exception of the average number of citations 


per paper, the partial indicators seem to suggest that the Lick 


3-metre telescope has been the most successful of the four, follow- 


ed by the Kitt Peak 2:!-metre and the Cerro Tololo 1.5-metre, with 


the INT a considerable way behind. In terms of size, the scientific 


performance of the INT_might be expected to be intermediate det- 


ween that of the Lick and Kitt Peak telescopes. In fact, it falls short 


of this by.a factor of six.in.terms of numbers of papers published 


over the decade, by.a factor of seven in terms. of the total number 


of citations gained in 1978, and by.a factor of up totenin.terms of _ 


highly-cited papers. That the INT should appear to be so far behind 


the Kitt Peak 2.1-metre telescope (itself never more than a: ‘second- 


division’ telescope) implies that its. contribution to. scientific 


knowledge has, in world terms, been rather small. If so, the INT 


has fallen well short of the claims made for it before it was built 


(and used to justify the major capital investment). This conclusion 


is shared by the great majority of the British astronomers whom we 


interviewed. Their disillusionment i is mirrored i in the rapid fall-off 


in the demand to use the INT; when it was completed, it was over- 
subscribed by -a factor of four,» but. by. 1977 *his had fallen_to 


under 1.3, and our evidence suggests that, at tira. the telescope 
was €ven undersubscribed.§-- 5 eee 

- Before moving on to > consider possible reasons for. the relative 
lack. of success of the INT, it is worth examining two other possible 
justifications that have-been given in the past for its support: that it 
has proved a valuable training ground for young British 
astronomers; and that it has been an important test-bed for edvanc- 
ed astronomical instruments subsequently used on large overseas 


telescopes: To investigate the first, we asked the views-of professors 


at four of the main university groups of INT users. All were scep- 


tical of the value of the INT for educational purposes, suggesting 


that nothing is more disillusioning for young astronomers than to 


have their allocation of valuable observing time on a large telescope 


ruined by bad weather. They also reported considerable instrumen- 


ial difficulties (one of the spectrographs, . in particular, yielded 


unreliable results) which made it difficult for students to use the 
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telescope. 5? Overall. it it's seems ; doubtful whether the use of the INT 


for. training_ astronomers tO operate a large seIESCORE constitutes 


suffi icient justification fori its existence. 33 


substantial. The most successful piece- of optical astronomy i in- 


. Strumentation produced i in Britain over the recent past — the Image 


Photon-Counting System (IPCS} — was developed-in a university 


astronomy department, rather than.at the RGO. Although tested at 


the RGO, the small 30* telescope (rather than the INT) was used 
for its development, while.the first astronomical observations with 
it were made at the Hale Observatories in California. According to 
the (PCS’s designer, the INT,.-while sed-for- some Subsequent 
observations, played no , Significant role- in _this _important in- 


strumental arguments - for the INT had been as strong as some 


would claim, * we would still contend that. investment in what was 
for-several-years Britain’s principal optical astronomy. facility. had 


to be justifiable primarily in scientific terms. Our results s suggest 
that, compared with other similar-sized telescopes, the INT proved 


relatively unsuccessful in this respect. What reasons can be found 


for this? -- =e 
One possible Siphaiiattoni i is that those astronomers using the INT 


were not as competent in undertaking significant research program- 


mes as were their-American counterparts. However, there is much 


cause to doubt this. American astronomers were in fact quite com- 


plimentary about British successes with_ the. Anglo- Australian 


Telescope. Moreover, the many British astronomers. (including 


some from the’ ‘RGO) _who have observed overseas have shown that, 


once given access to large telescopes on good observing sites, they 


are no less competent than other astronomers... _ __. 


SIS EE a 7 Sa aa a a 


_A second common argument _is that the INT's. relatively poor 


performance. _was due _to inferior. observing conditions in Britain. 


There can_be no dispute that the RGO site proved totally un- 


suitable, confirming the worst fears of the few astronomers who 


warned against building the telescope in Britain before construction 


began: The site is at far too low an altitude — a mere 30-imietres 


above sea-level, compared with several thousand metres for the 


best. mountain sites. - The humidity is often high (the RGO 
overlooks Pevensey Marshes), making-frequent recoating of the 
telescope mirrors necessary. The street lights at a nearby town tend 
to illuminate the night sky. The INT was at one of the most 
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Northerly latitudes of all the world’s lates iebeseopes: in summer, 
when-the weather is best for observing, the nights are rather short. 
Finally, and perhaps-most importantly, the weather in-Britain is far 
from good, with a high percentage of cloudy nights. * 

The crucial question, then, i is whether. the site alone explains the 
factor (somewhere between six and ten) by which the scientific per- 
formance of the INT lagged behind that of equivalent telescopes 


overseas. According to the RGO there are, on average, approx- 


imately 1,200 hours of clear night sky at the site.*’ This is only a 


“factor of two les$:than the 2,500 hours or so obtained in the South- 


West. United States at. Lick_and_Kitt Peak Observatories; this ap- 


parently leaves.a factor of between three and five still to be explain- 


ed. The number of hours of clear sky.is by no means the only factor 


determining the astronomical observing conditions: but even if one 


allows for. the additional effects of. poor.‘secing” conditions, low at- 


mospheric transparency, and the fact that.a certain proportion of 
the quoted 1,200 hours of clear sky. comes in the form of short 
periods (of little use for many types of observation), it is still-uncer- 
tain whether the remaining factor can be completely accounted for 
—- particularly since the Lick Observatory site suffers to a lesser 
extent from some of the same problems.  - 

_ Many. of. the astronomers we-interviewed were actually rather 
doubtful. whether the. INT's relatively. Poor. performance could be 
was One major cause. They. expressed concern with the way in 
which the INT and its associated instrumentation had been built 
and = and cited a ‘humber of what they perceived to be 
iciilaé, clearly felt strongly. Those most frequently. inenitioned i in- 
cluded relatively poor- instrumentation -(particularly the original 
coudé spectrograph, which-was not. ready until several years after 
the INT was completed; and-which then proved unsatisfactory); the 
RGO’s claimed failure to fulfill its role as a provider of central ser- 
vices (especially technical back-up) for university users of the INT, 


serving university astronomers. were often t io! busy r meeting the r re- 


strumental. innovation. at_the RGO. compared with other major 


observatories overseas (the main reason for this being given as the 


isolation of. the technical. development. work at the RGO from 


university astronomers); and a difference in approach towards the 
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demands of research — - university. astronomers. contrasted what 


they termed the ‘civil service’ mentality of RGO staf f with the ap- 


proach found in their own departments (citing, for example, dif- 


ficulties they Snpowitered in attempting toc callout RGO staff for 


assistance after hours). However, ttiese-criticisms aiz not accepted 


by most RGO staff. Although some admit that there may have 


been problems during the early years. of the INT’s. life, when-the 


RGO was only just beginning to a adjust tio its new role as a provider 


of national_facilities, * they argue-that significant advances have 


been. made over the last few years. In their view, the perceptions of 


university astronomers have not reflected these changes, and the 


criticisms should therefore be seen as referring to a much earlier 


period in the INT’s history. 


There are thus two principal interpretations of the dift ference bet: 


ween the scientific performance of the INT and that of similar 


telescopes overseas: According to the. fi rst. — the one- favoured ai 


the RGO — the poor performance of the INT is entirely due to the 


site. In contrast, according to a large section of the astronomical 
community outside the RGO, the site-only explains-part (albeit-a 
major part) of the difference; the remainder is related to the \ way in 
which the RGO discharged its responsibility for the support and 
operation of. the telescope. This latter interpretation appears to 
have been the on: favoured by a majority of the Northern 
Hemisphere. Review Committee at the time of their report.” 

. In the light of the INT’s comparative lack of success, can any 
Policy conclusions.be drawn? First, -subsequent-events imply that 
the decision to build a-2.5-metre telescope_in Britain was almost 
certainly wrong. For the same amount of money, a somewhat 
smaller-telescope could have been built on a good site overseas. If 
this had been run-by an organization similar tothe group of univer- 
sities operating KPNO-.and CTIO, then, in the-opinion- of many 
astronomers, it would-have proved as successful-as the Kitt Peak 
2.1-metre telescope.(which isabout as far from the universities of 
some of i its users as the La Palma site is from Britain) and the Cerro 
Tololo 1.5-metre instrument.(which is far more remote). Hf the deci- 
sion to build. the INT in Britain was wrong, could this have been 
foreseen, aiid was there any alternative? The answer to the first of 
these questions is, "Yes, almost certainly, if. sité- testing of the type 


the second, many astronomers are now under the impression that 
the British Treasury and Admiralty, when they agreed to fund the 
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INT in 1946, would not have consented to an- overseas site. 
However, we can find no evidence in the relevant public documents 
for the late 1940s to support this view. * Instead, all the:documen- 
tary evidence indicates that the British site was chosen in 1946 by 
senior astronomers, who produced various arguments to justify 


muronomes. that. the Treasury might object to z an overseas Site, 


long after the original decision on the site appears to have effective- 


ly been made. The threat that the Treasury_might withdraw its 


backing for the project was then_used to dissuade those younger 


astronomers who harboured doubts about the value of a British- 


based INT from expressing them in public. == 


_ The second set. of policy implications concerns the decision to 


give t the RGO. responsibility for the new La Palma Observatory. If, 


RGO's. handling of the INT, particularly with. respect to their 


responsibility for providing modern instrumentation for the 


telescope, was indeed one of the reasons for the telescope’s_com- 
paratively poor scientific performance, then some other justifica- 


ding over responsibility for the new observatory to the RGO. 


However, because the report of the Northern Hemisphere Review 
Committee (NHRC) was not made available to the wider scientific 


community, there was no public debate as to whether such a 


justification actually existed. = = __ Bee. 


_ _Was there an alternative to making the RGO responsible for the 


‘Ea Palma Observatory?. The NHRC certainly believed that there 


was: They argued forcibly for a ‘third centre’, funded by the SRC, 


similar to_that operating the national astronomy facilities in the 


United States. But were there scientists of sufficient calibre to staff 


such acentre? Again the NHRC had no doubts: -In addition. to. cer- 


tain. astronomers_in_the United Kingdom, some of whor have 


subsequently. enjoyed great. Success. using. the Anglo-Australian 


telescope and other major. instruments. -overstas, there were many 
British. astronomers working in-the United States, some- with much 
distinction, who at the time expressed a willingness to consider 
returning to Britain-to take_part in such-a.venture. The impressive 
nature of. their track-record would ‘certainly have argued well for 
comparative data on the track-record of the INT were not available 
at the time. If information of the type described in this paper had 
been obtained at some stage between the completion of the NHRC 
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report and the eventual decision i in 19741 to make the RGO responsi- 


ble for the La Palma Observatory, then surely that report — and in 


Particular the recommendation_of a new independent centre to 


operate the observatory —- would have received some public discus- 


Sion, and its merits and. drawbacks would. have been openly assess- 


ed. The provision of information.o on the INT’s performance up to 


1974; even if in the last_resort it had not led to a different decision, 


would at. least. have_resulted. i in.a more informed decision. Our 


results also appear to provide ‘support for recent moves-to ensure 


that the operation of. the La Palma Observatory allows i increasing 


SCOPE, i in its control. and management, to university astronomers — 


in line with official SRC (and SERC) policy for the past fifteen 
years. © 


® NOTES 


The authors wish to eekiionteige i the support of ihe UK Social Science Research 


Council (grant No. HR $175/1) in carrying out this research: They are also grateful 


for helpful comments on an earlier draft. of this-paper from various- colleagues, 


astronomers and referees, and for a significant editorial contribution from David 
Ear. 


: No order of seniority implied (rotating fi fit rst authorship). 


. The British Treasury approved plans for a-100-inch eens but the INT 


eee in fact built with an aperture-of 98 inches, slightly smaller than the 100-inch 


reflector-on Mount. Wilson, in.California:. By the time the INT was completed, 
another large (120-inch) reflector-had-been completed in- California: (at- the Lick 


Observatory), so the INT was then only the fourth largest telescope in the world; 


however, -it remained the largest in Western Europe until the late 1970s; when it was 


dismantled and moved to the Canary islands. For a detailed history of the INT. see 


F.G.-Smith and J.-Dudley,The Isaac Newton bases Journatioet the History 
of gee Yor 13 (1982), Rt 


380-83. = 
4. B.R.-Martin and J. levine: - ‘Assessing Basic Research: Some Partial 1 In- 


dicators of Scientific Progress in Radio Astronomy‘, Research Policy, in-press; Mar- 
tin and. lrvine, ‘Internat - -Criteria - -for.. Scientific Choice: An ua ae of ane 
press. We will refer to these papers, below, as (respectively) Paper 1 iid Paper 2. 

Where page numbers are cited, they refer to the mimeo versions obtainable from the 


authors. 
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S..Cf. J.. Irvine and B:R- Martin; “What. Direction for. Basic | Scientific ic 


Research?’ submitted. to Nature. For recent discussions of peer_review, see 1.1. 
Mitroff and D-E. Chubin, ‘Peer Review at the NSF: A Dialectical Policy Analysis’, 


Social Siudies of Science, Vol. 9 (1979), 199-232; S. Cole, 1.R: Cole and G.A. 
Simon, ‘Chance and Consensus In Peer-Review’, Science, Vol..214 (20 November 


1983), 881-86; and D.R. Hensler, Perceptions of the-National Science Foundation 
Peer. Review Process: A Report on @ Survey of NSF-Reviewers-and Applicants 


(Washington, DC: Nasional Science: Board and Committee on Science and 
Technology,-US House-of- Representatives, December 1976). - - 

. 6. The general implications of this for resource allocation in basic research are 
considered-in-Irvine and Martin, -op. cit. note 5. For a brief discussion of ‘research 
Oligopoly’ in British radio astronomy, see D:O. Edge and M-J. Mulkay, Astronomy 
Transformed (New York & London: Wiley-Interscience, 1976), 250-53. One possible 
consequence of the development of an ‘oligopolistic’ structure in Big Science is that 


funding decisions can-at times appear to-depend less on criteria relating to scientific 
merit than on. political factors associated with the defence of institutional interests: 


it did not surprise us-that several-of the British astronomers we interviewed com- 
plained about what they saw as the contro! of research council. commiuees by a 


secretariat who seemed to feel a greater responsibility towards SERC establishments 
than Co university scientists. (The UK Science Research Council [SRC) was formed 


in 1965; it was renamed the Science and Engineering Research. Council ISERC}.in 


Selick centre may support-applications for hinds: by other centres — even if it 
means failing to evaluate critically proposals of dubious scientific meiit — because 


they recognize that it is in the common interest-of all the centres in their specialty to 
maintain a united front vis-a-vis alge 2 agencies. And, indeed, ox we interview- 


However, detailed examination of such elaimns i is beyond the scope of this j paper, ar and 


pressures are present in most Big Science specialties, in most countries: the only ex- 


ceptions would appear-to be in cases where research councils seek the-help of inter- 
national experts.in- order to obtain impartial (or,-at-least, less partial) advice. For. 


farther discussion of these issues; see: D.S: Greenberg, The Politics of Pure Science 
(New York: New. American Library, 1967), published. in Britain as The-Politics of 


American Science (Harmondsworth, Middx.: Penguin, 1969); M. Blissett, Politics 
in Science (Boston,-Mass.:-Little,- Brown, -1972); M. Mulkay, "The Mediating Role 


of the Scientific Elite’, Social Studies of Science, Vol. 6 (1976), 445-70; S.S. Blume. 
Toward a-Political Sociology of Science-(New York-and London: The-Free Press, 


1974): and C. le Pair, ‘Decision-Making on Grant Applications ina Small Country’, 
Scientia Yugoslavica,-Vol.-6, Nos-1-4-(1980),. 137-43. -. ---- : 

__ AS to decisions in British optical astronomy in pariicelar; the sitanion. is com- 
plicated since, from the-late 1960s (following the formation, and developing in- 
fluence; of the Science Research Council), official policy has been to develop the 


central; government-financed scientific facilities (such as the Royal Observatories) 
with more specific reference to the needs of university scientists. It was intended that 


the- jatcer should Diay | an increasing role- in the planning and —— of these 


below. The sojourn of the INT at the RGO voineided with-the sadaal implementa- 
tton of this policy, when the RGO was ‘in transition’. This papér can be read as a 
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comsbeiies to.a more pointed agenda for future research and discussion on these 


aspects « of UK science ‘Policy. _ 


---9,-A.M. rps caaes a ae for Scientific Choice’. Minerva, Vol. 1-(1963), 


159-71: This paper led.to-a debate, the main contributions to which have been col- 


lected together -in E: Shils (ed:); Criteria for Scientific Development: Public Policy 


and National.Goals (Cambridge, Mass.: The MIT Press; 1968). 
10. Weinberg. op: cit: note 9, 163. 
1. Paper-2, op. cit- note 4,12. _ - 
12. See Papers 1 & 2: Op. Cit: iiote 4,. aiid iste 8. 7 


13. For a discussion of these terms. -see Paper 1, op. cit. note 4; 15-17: 
14. The details of how the main.problems with the various partial indicators are 


overcome in the-method of-converging partial indicators can be found in Paper 1, 
op. cit: note 4, Figure 2, 25. 

- 15. See J. livine and B.R. Martin, ‘A. Methodology for Assessing the Scientific 
Performance of f Research Groups’, Scientia Yugoslavica, Vol. 6, Nos 1-4 (1980). 


83-95. Further ‘Comment is perhaps necessary here on our use of the term eane 


know ledge c coming from.that telescope, regardless of -whether it was used by the staff 


of the centre responsible for its operation, or by visiting users. The research perfor- 


mance of a facility therefore « obviously depends o on its use by-both internal and exter: 


nal users, as-well as on its quality, size and: vinta; and the instrumentation and 
back-up services provided by the central staff. - 


- 16. The RGO and-the Royal Observatory at Edinburgh have ¢ together accounted 


eames and subscriptions ¢ to various telescope facilities overseas. 


ie Hens Abt, ‘The Cost-Effectiveness of Telescopes of Various Lalgaisied : in 


assembled -by-one of us for a forthcoming: doctoral thesis: B.R.-Martin, The Evotu- 
tion.of. British Optical Astronomy, 1945-75: A Case Study.in Science Policy (PhD 


thesis, Department-of Liberal Studies in Science, Manchester University, in prepara- 
tion). This thesis draws heavily-from-Public Record Office documents, and other 
publicly available material, as weil as interviews with certain Senior astronomers. See 
also Smith and Dudley,-op..cit. note 2. 3 

19. In 19$8, the British Government temporarily withdrew its backing f for the 


project, presumably exasperated by the long delays-in settling the design, and by the 
soaring costs (by then some three times the Original estimate). This delay, however, 


accounts for only one year out of the twenty-one it took to complete the telescope. 
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20: For details; s see e Smith. ad Dudley, Op. Cit: note 2. 


--21. H.H. Plaskett, °Presidential--Address’, Monthly Notices of the Royal 
Astronomical Sociery,. Vol:. 106 (1946); 80-94, Sp. 95. 


- 22. Of course, British-astronomers always-knew that, in terms of ‘seeing’ condi- 
tions, there.were much better sites abroad, bit it appears that, in the mid-1940s, 


many simply did not press to establish whether these were real alternatives; ‘open 
options’: See note 33 below, and the ‘Discussion and Conclusions’ section of this 


paper. As R.O. Redman (one of the strongest advocates of overseas sites) put it in 
1946, "Astronomers acquiesce, becatise the choice is between-a new-instrument.in a 


bad situation or no instrument at all’: Observetory: Vol. 66 ( 1945), 153-59, quate at 
157, = 
- 23. This-figure of £7.5 million: for setting up the int on ba Pama now includes 


the cost of certain central facilities that -will- be -used -with the new 4.2-metre 


telescope; as well as with the INT. Even so; some of the astronomers we interviewed 


felt that this figure was. rather -high,- and suggested that-a new, purpose-built 


telescope-could have been constructed for the same price. They argued that, since 
the new.4.2-metre-telescope will-cost some-£9. million, and since the-capital costs of 


telescopes vary approximately as the 2:4 power of the aperture (see H:. A. Abt, ‘The 
Cost-Effectiveness of Telescopes of- Various Apertures’ (Tucson, Ariz.: Kitt Peak 


National Observatory: mimeo,. 1980}); such a new telescope could have an. aperture 


Of well over. 3 metres -— that-is,-appreciably greater than the INT’s 2.5-metres. In 


Contrast, other astronomers felt that the costs of the INT’s move were justified: they 
seemed to welcome this opportunity (as they saw it) of, on the one hand, finally 


removing the embarrassment of a major.research facility that was inevitably under- 


exploited because of the British climate and, on the other, leaving RGO staff free to 
concentrate on the new facilities on La Palma... 
24. R. v.d. R.Woolley, as reported in Nature, Vol. 180 (31 August 1957),-420. 
25: This is now known as the La Paliia. Observatory, or the Roque de los 


Muchachos Observatory, and will consist of the-INT;the 4.2-meitre telescope-refer: 
red.to in note 23, a 1-metre  ESCOp the iin eae disk, and a number of 


smatler-telescopes. 
.. 26. For example, see G. Burbidge, Correspondence, Nature, vol. 239- (3 


September. 1972), £17-18; P.A. Strittmatter, Correspondence; Nature, Vol. 239 (20 
October 1972), 475. - 


- 27, Others claim that this already effectively: happens; through ¢ the RGO; follow- 
ing the change in policy starting in the late 1960s (see note 6). 


- 28. Of the nine Committee members:and consultants, just two (ine Directors of 
tie two Royal Observatories) dissented from.this recommendation, which was itself 


an aspect. of-a general shift in central policy (see the end of note 6): It should be 
noted that this information, as-well-as later discussion on the role and reocommenda- 


tions of the Committee; has been drawn directly from. the Report of the Northern 


Hemisphere Review-Committee. Though-this. was never published by the SRC.-we 
were given access to the Report by one of the members of the Commitee: A detailed 
historical analysis of decisions in UK astronomy policy since the mid-1960s will ob- 
viously have to consider the e srecile composition of this  Cominites, ane of maker 


related _b sdies. 
. 29. Se, in particular, * "Scandalous Mudie i in- British Astronomy’. (Editorial, 


Nature,-\-ol. 239 (18 September 1972), 121-22. It may have been partly in reply to 
the implie ation contained in the Northern Hemisphere Review Commitiee’s report 


r) 
ERIC 


11 


irvine aiid Martin: The Isaac Newton Telescope 83 


(which, alihough unpublished, was the subleci a well-informed iesen i that RGO 
astronomers had been tess than successful as research. scientists,-that the-Science 


Research Council Claimed that * *the scientists working i in the e SRC Establishments’ 


in different fields*:-UK- Science Research Council; Annual Report for the Year 
1971/72 (London: HMSO, 1972),-$2 (our emphasis). Neither side, however, produc- 


ed-reliable data to back up their assertions. 
30. From our interviews, it is clear that there-was considerable resentment within 


the scientific community about the. way the Committee's report was treated. The 


report. was never.made public, so-denying the main body of the astronomy com- 


munity the-possibility of open discussion of its analysis and contents. indeed. ihe 


Teport was kept.so-secret-that one of the members of the Committee that had drawn 


icup informed us that, when he requested a copy, hie < 23 told that this was not possi- 
ble because its contents were ‘confidential’. 
31. See Paper I, op. cit. note 4: 
~32. There are several other telescopes of similar size; like the Sieward ¢ ise 


vatory 2.3-mietre, and the Texas McDonald 2.7-metre arid 2.1-metre telescopes, but 


these-are all at essentially single-university observatories, while the four nlelreere 
included_in our sample are. multi-university ee 


snes, in 1946, -to build a trae telescope. 9 Howe 


the documents for this period did they ascertain whether ¢ OF not the Treasury. would 


agree to an overseas alternative. For further details, see Martin, op. cit; note 18: 


(Telescopes sited abroad obviously cost more to operate, so a slightly -smaller 


telescope would have had to be built in order to keep within the INT’s budget. ) 
34. See Abt, op. cit. note 23. 


35. It must-be emphasized that the figures: in Table 2 are fmiended to mindicare Wt al 
costs, rather than | marginal or direct-costs. Abt (op. cit. note 23, 15) quotes « figure 


of $173,000 (£74,000) for the direct costs of the Kitt Peak 2.1-metre telescope in 1974, 
at the same time pointing out that this represents only a small-fraction-of-the-total 


costs — it corresponds to one quarter of-our estimate of the total costs: Similarly, 
material provided by the RGO-suggests-that the marginal-costs-of-the INT in 1972 


were about £100,000; again approxknairly third or a bavariee of the toral etek as 
estimated.in Table2. .- — -- 


36.- In his calculations, Abt (op: cit; note. 2B) uses & figure of 7 7S years, bui this 
would make little difference to the total-costs in-the- final row of Tabie-2. 
37. If one allows for the effects of inflation, the figure would be rather baghiet: 


- 38. This is is not to say i users ¢ oft the INT w were t satisfied with t the funds available 


counterparts. This is consistent with the wen expressed in the Atlas Hemisphere 


Review Committee's report that the RGO ‘enjoys resources large in comparison 
with most observatories anywhere in the world’. 


39. In the particular case of optical astronomy; researchers often use published 
papérs.to account for the periods of observing time they are allocated_on. major 


telescopes. This is probably why Abt (op. cit. note 23) finds that the number of 
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publications in in any one year from each Kitt Peak: ideacone conesponde s approx- 
imately with the number of users-of that telescope in the previous year. 


40. See, especially; Paper 1; op. cit. note-4. . 


41_ These increases are not without significance. for policy-makers. In our study 


of high-energy physics electron accelerators (Paper 2; op. cit: note 4); we found a 
rapid dectine in the numbers of citations to-recent work after the first few year$ of an 


accelerator’s-life, suggesting swift obsolescence of such machines as frontier- 


research tools. In optical astronomy, the reverse appears -to -be true: optical 


telescopes seem to become more productive with time (at least for the first ten years 


or more}, presumably-because better-instruments are being fitted to them. This con- 


clusion is in line with the Jong working life (fifty years or more) of telescopes, refer- 


red-to-above; in contrast, an accelerator has a-useful working life-of perhaps-ten or 


fifteen years, after which it. normally CEASES LO Contribute significantly to the ad- 


vance of high-energy- physics, and is soperseded-by more powerful accelerators. 
42. Those readers tempted to reply ‘The British wembe" are referred to wur later 


"Discussion and Conclusions’ section. - 
43. Peer-evatuation; in contrast, should provide some soit of. weighted average 


for both-the great mass of relatively minor contributons made by a-telescope, and 
— Small number of major discoveries: see Irvine.and Martin, op. cit. note 15, 86. 


. In doing so, of course, we are temporarily shifting the focus of our quan- 
dais evaluation from the single telescope at t each ¢ centi= tncloded is in our  sttedy, | to 


_ 45. Kitt Peak astronomers ranked tt themselves - 1 9 compared with 2. 2.6 given by 


other. -astronomers; Lick; 2.3 compared with 4. is and RGO users, 10.4 a compere 
with 11.0. - a 

- 46. In 1977, a total-of approximaidy 140 papers were re published us using siscivas 
tions obtained on Kitt Peak telescopes — about twice the number. produced from the 


various Lick telescopes. Using Abts finding (op. cit: note 23) that the number of 


citations earned -by- Kitt Peak--telescopes of different apertures. varies as the 
1.$-power of ‘the ereniers one can estimate ‘approxin.ately the total number of cita- 


obtained: | In 1978; the coud number of citations would probably have been over 
3000, compared with well under 2000 for work from all the Lick telescopes. - - 
.47. These two centres were not included in the original list that we asked 


‘astronomers to rank. -However, several of those interviewed mentioned these two 


observatories, generally placing them oa Cerro Toe 2 and alicad of the 


University of Texas ‘McDonald Observatory. - 


we ar felt it necessary_to devote so-many-pages-to reaching it. As one referee put 
it: ‘Asse:sing the cost-effectiveness of the INT is like being asked to prove that it is 


colder « the poles than-the equator’. However; having clearly summarized 
astronomers’ opinions on this matter, he went on: ‘but opinion does not. persuade 


the powers that be; and cozsequently an exercise suct. as that 1 represeniad by this 
paper is necessary for that. purpose’. - 


- $0, That is to say, astronomers applied for four times as many nights of ebueiv: 
ing time as were available. 


ll 
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51. Evidence [ for this came from a an interview with a former member of the SRC 


Panel for the Allocation of Telescope Time. 
_.52. The head-of one major niversity astronomy. department recalled: "We tried 


ble to get any reliable data. We never ied a again fo: 


: 33e —One senior ‘astronomer commented: ‘For. many. years, Sir R ‘Richard 1 Woolley, 


Previous big telescope experience — a chicken am exe situation’: 
$4. While the INT has been used-to develop various instruments and iechaness, 


it is by no means: clear whether it was essential for their development. Several 
astronomers. stated. that they -used the INT for- instrumental development solely 


‘because it was there’, and because it was easy to obtain time on it: without the INT, 


the development woud, they believed, have gone ahead on other telescopes, as it did 


toa pigs extent with the IPCS. 


off from-the INT has been small compared wi — peice investment involved, and 
also small compared. with the spin-off noted at other Big Science centres in Britain: 


see J. irvine and B. R. a "The Economic Effects of Big ‘ Science: The Case of 


fects of Space and Other ace Technologies, Strasbourg, . 23-30 April | 1980 
(Paris: European Space Agency, tef. ESA SP-ISE, 1980), 03-16. ; 


56. - Just how unsuitable the site is could have been. established long before con- 
struction of the telescope began in.19S9. Perhaps most senior British astrononiers 


thought such a pedapengened test otiose — a matter of “proving | it is colder at the 


jest between 1973 and 1978. 


whether our respondents’ pt ny of these *problems* are and accurate — 
that, as we have already said (see note 6), requires further-careful, detailed and 


specific rezearch, and is beyond the scope of this Paper. We are merely staring that 


these Claims and assessments represent the feelings of many university astronomers. 


- 59. See the end of note 6. As-one senior astronomer, with experience in both 
universities and Royal Observatories,-wrote to us-latein 1981: ‘The nove from con- 


centration on in-house research t~ the provision and operation of national facilities 
in the fate- 1960s and. early-1970s involved a profound change in the Ro: 


vatories, and I think it is true to say that (o.a greater or lesser extent (hey resisted or 


went-along gruagingly with the-new regime. J-know for-a fact-that there are still 
members of the Observatories who believe that the current role of the Observatories 


is-wrong.and that the old system was much preferable.-However, the new role of the 
Observatories is now firmly established and the staffs of both Oteepralorics are 


respcnding with-enthusiasm to the new challenges.’ - - 
GO. See ole 28: as we state there, further research into-the composition of thes 


Committee, and into-its findings, is required. 
61. We have examined all the relevant documents av: ailable a at ihe Public Record 


Office, the Ministry of Defence, and the Treasury. We-have seen nothing to indicate 
that the expenditure of the same sum of money on a similar telescope overseas was 
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not an. open ance in 1946: For a 5 comvaie interpretation of these e events, 3 sce 


Smith and Dudley, op. cit. note2.- —- 
62. Currently one such expatriate astronomer is Director of t the Kitt Peak Na- 


tional Observatory, another is Director of the Steward Oboservatory (one of the mest 
successful university facilities — see note 47), a third has been one of the most 
distinguished users of the Lick 3-metre telescope, and-a fourth has figured pro- 
minently in the achievements of the Hale Observatories (by our Beer-evaluation, the 
world’s premier optical astronomy centre). -- ee 

63. See the end of note 6. ‘The so-called ‘Tiger Team’ set up to examine the RGO 


plans f for the new 4,2-metre telescope contains-a strong representation of university 
astronomers: this is another indication of the progressive development-of this 


policy. Another is the fact that-both the:RGO and the Royal Observatory, Edin- 
burgh, now have active young Directors who were previously university astronomers 


with international reputations — rather than civil service scientists already ‘within 
the system’. 
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Assessing basic research 


Some partial tndicators of scientific progress in radio astronomy 
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AS the conts_of certain types. of. scientific revearch have 
escalatert. andl as growth rates in ewerall aaiiomal scrence bredgeit 
have declined, sn the werd for _Seleaphecit science policy has 
frown more wrgcet, rie 


aaah: hs) ieee at Pm ats ‘ 


indicators, only im those cases. where 1 id 


tesearch grenpre crmpeting For scarce fends. one of the mou 
imperient pieces of information aceded. hy ecience pobscy- 
makers 6 an mscsiment of thoee grou’ recent eieniific 


perloemance. This paper mggeus a method for evabeming thet 
perlocmence. . 7 
“Alter seviewing the Weratare on seenillc attcpemrnt, we 


aumber of" pertial itestors’ ~ that-ts, variables 


grenns han -hecn largely, smccestfal). and hat the: | inienicore 


therclore provide a ressemably rcliahle estimate of the cimtribw- 


then to ecicmific pengress made by diflercat renearch Reems, - 


ees. “the methed ef crmverging fiettial imdicaines is tevied. aad 
several of the indicators (publications per revarcher, crtatiwe 
Per Paper, cumbers.cl highly cited papers. and rece evahean we) 
are found to give Initty comiisicnt sewlis, The rewlts are al 


eelevance te te questions: san havc 


determined 

partly hy the magnitude of the perticular conirthation:. and 

parity hy “other factors’. Ef the parilal lndicsion... : ia yack! 

sctahle revels. shen the infiuenee of thaw “otter factors” wren! 
ks bt G = 


(h) more specifically can significa 
performance-of raden stiremnmy ecntres he hed? 

would meintaia that the evidence presented in this paper is 
sufficient tn justly 2 positive antwer to hath these quesiinns, 
and heace to sinw methnd af converging partial 
Biomass wreful in science policy-makers 


citations ace seem aa reflecting the impact, cather than the 

qeality ov impretence. of the rescarch work: 13) a renge of 
lndicsters ae employed. cach of which focanes.om diferent 
2 (4) the indicators are applied 


fo maiched growps, comparing ‘lite’ with ‘he’ as (ar as 


> cde often epg Ns Stig th 
payee e currently Fellows of the Science Policy Rewearch 
Unit-ehere they work-on 9 range of -isseet onmaccied with 


mn Onnwhey withes tn dimnciate hirmell crenplerety 
eee apne. 


Revearch fh Policy 12098) re 
North-ttiend Peblishiag Conpeny 


i. inwodection - The aced for an assessment of 
baste research 


_ Few would now dispute the ® sestity for soitie 
assessment of the output [rom basic research. As 
the scale of basic scientific. research has grown, se 


icy that can only be 


policy that can « 
$ onstiucted on the basis of an evaluation 
of the _output_ from that research. When_ basic 
research was little more than _a hobby for gentle- 


folk. and. even. ax fecently | as 3 years ago when the 


questions concerning the organization and Funding 
of science to scientists. There was no need for a 
Policy establishing priorities across the sciences, 


for for an evaluation of the performance of scien- 
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42 ALR Marue and J. (reene / Avcvimg have rerereh 


Icft to the «cientists fa amis Now, ihewever: the 
resmurces devote so hasic tesearchia “are so wilgecakeg id 


Science Resench. Councit- (SRO)! for the financial 
year 1928-1979 aimounied in well over C150 mil- 
ten. Although some of this was allocated to en 


went = that is, res 
search canied out with the primary purpose of 
increasing scientific knowledge rather: (han crcat- 


jiaeey lor basic research, and 
hence for analytical tools to assess such reseatch, 
has heen grenily sl 


cconamic recession in the 1970s. 


lised 
nations have been forced to look more carefully ‘at 
all-carcas of. st. te expenditure. Whereas _in_ the 
1950s and 3960s econcmic growth sales. generally 
permitted, public expenditnre.in one ares such as 


fe ng other ar nin 
Crease in one area of public expenditure tends only 


and who argue for continued real increases in their 
level of funding, must now be willing to accept 
wider public examination of their research work ini 

order tliat one may “asceriain what benefits are 


ae project _promises cerisin benefits or 
“good science”; instead, it mtxt he Semonstrated 
that the Project | is . likely to . Yield, renter benefits 


them in one pera ort science nro 2 
For policy-makers, there are four main seis of 
declskons thal need to he made In the allocation of 


to. he. Possible . at ihe expense of cut-backs in 


resemices to hasic research. 


(1) Hew much overall showk! be spent nu basic 
research compared with other areas of public 


(2) 0 


(3) How much should he ter the different 
types of scientific institutions? In Nritain, for 
example, what, proportions should be speni on 


esiahlishments, and on university research? 


(4 Mow much should he altocated to each re- 


search centre, group. of individual, working 
_witbin adi a 


rescuirces to scientific research should be based. 
distinguishing between “internal™ and “external” 


tif field itself-and answer. vhe quest 


well is the science done? ct Bate ster ja_are 


criteria to decisions about sci 
hence-to focus on the first: two of the four ques: 
Gons identified above. Mucit attention was Biver 
to such factors as the contributions of ‘science in 


and governments have in recent year ea tc 
take account 

arriving at decisions on funding. However, interna’ 
criteria cannot be ignored, | particularly in in: the cast 


inthis paper, we present. a. hamework - fo 
assessing the ‘relative contributions -to- scientific 
knowledge made by different research groups Ir 
the same discipline. This has been developed: a: 
part of a study of a small number of major 
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basic: research “eentres. ‘ The niain reasini - “for 


tentist is that over half 
the annual SRC expenditure gocs ta support re- 


f] 
In. provide inlormiaiion which, may facilitate the 
taking af decisinns of the third and. in particular, 
the fourth type identified above: In deciding the 
distrihutinn of resources ‘hetwecn fesearch centres, 


ane al the 


‘is hy no meane the only factor det 


future reacarch performance (there are aulices like 
ness” or potential for expinitation of new 
work); hut iis undeniah ly one of the most im- 
portant. other: factors arc equal, Of alternatively 


On average. a new scicntilie pre 
is more likely to be carricd out well by a group 


with a record of successful-research-over the recent 


We are concerned in this paper with the question 
nf_the extent to which: past performance can he 
ly and satisfactorily evaluated. The questions 
to which we have addressed ourselves iake ihe 
Tollowing form: to what extent can ihe outputs of 


whether one research group has contributed more 
to scientific knowledge than another, and. if sa, 


how? 
_ The ‘Steuctare of she pape is-as follows: the first 


progress can yield i ir. icgnsiion.e on the relative per: 
_ foemance nf groups of scientists: the later sections 
then detail our cinpirical: work i in wuich several 


forninnce nf a number nf Varge, hasic-rescarch 
centres working in the fie’s ni radin atranomy. 


2. The nature of hatte ¢ research and its outputs 
- While assessing the outpul from. hasic rezearch 
may be extremely difficult, there ¢ can be no douhi 


of new scientific knowledge (1 (heo- 
rics, empirical findings, and so on). new-scientific 


problems, or-new practical ideas or probiems (cl. 


Freeman [14)). In. short, there is a Now through 
hasic science of information generally embodied in 
research gee ofr in people. Rotrawing a 
| vocabulary of 


conceptual as. well as methodological - problems 
associated with finding appropriate: output. mea- 
sures. These stem from he intangible: nature of 


* The project aimed io-exseas_ihe perlormance_of five major 
centres supported by the SRC. These were the Daresbury and 
Rutherford high-ertergy Physics taboratorier, the | 


the optical astronomy fecifities a1 the Royal Greenwich Ob. 
_ Servatory. | 
? The “CERN,” Rutherford, Daresbury, Appleton. end IL.t. 


stat 
helter represented as a ‘series: of revolutionary 
transformations = cf. Kuhn. [25])..as weil as 


our approach to fhe sociology of scientific 


Crenohle Lahorstories, ingrther with the two Royal Ofterva- 
tories, acconinted for £87 miltien of ihe £187 miffien spent. by 
n= SRC te the Mnanctel yea: 197K. 79, while several other 
majen_tesearch groupe, Incheving the tadio asitanamy ob. 
servatories at Cambriige and Manchester Universiites, werc 
ate supported mainty through consctidated binck grants 
eather than ihrough grants for specific reseatch projects. 


jor 
tively small-incremental- additions: to our know!- 


edge (perhaps in the form of more precise mea- 
surements), Roth types. of contribution are. ~abvi- 


scientists who are primarily teachers, adeninisiea- 
tors. of technicians, all play crucial roles in scien- 
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ecientists 


_ eeanite 


de are “be _— uc to the primary peal 
af basic research. 


wlions ta scientific knowledge 


incivde the number of scientific. publications pre. 
Hueed in ina given period ‘or for a. given volume 3 


Scientifa prowbuction 


(1) Canceptval. capcesmentat, 
and technical eink af 


(2) Diewenination of research 


19) Seppart work by technicians, 


CH) Scientifar count rsbeeticnne 
te the singopline - mew _ 
mientilic knnwinige and 
tev * - ot 
(2) Scientific comicthations 4 
_ ether arene ol ecicnre 
(3) Edecational Comtrthetion = 


Cepia 
£5) Costiewst ereuribeatione 


plines: ands countries, oF i aver an_extended time- 


scale (cf. Freeman | 4. p._ 4). Itis the contention 
of Ahis:paper that, although na. absolute e Quantifica- 


id 
etul ¢ given hoth to the choice of 
groups. for comparison, and to the question. of 
what the various indicators of scientific perfar- 
mance are actually measuring 


lifie peers of the a of the pub- 


ished » work, the. number _of “discoveries” or. other 


icull_ in fas (and 
verhaps invalid theoretically) to extend. such: in- 


lices of performance across a wide range of disci- 


another paper (irvine and Martin [22)). This considers beth 
1 spia.nlf and the migration from radto_asiron- 
omy of trained stedents, who take with them stills shat 


suheequenity prove useful in a variety af high. technclogy 

ain focus of our work has been on 
fe Knowterge. hecaete nearly all the 
mrmbers of the research reupt whom we interviewed gave 
th primary justification for continued financial sup- 
heir research, rather than ether ovipuls such at 
contribetions to education on tn technotogy, 


yp UN 


sane progress 


their time-spent on research, and the number of 
support staff (for example, in the form of teclini- 
cians oF administrators).1 The second, 1 Fexearch pro- 


form of research communications. although sclea- 
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tists ht commnnicae through other informal 
channels ‘such ae Netterx,- seminars, and personal 
conversations. * The third, scientific Progiess. cc. 
tic tific activity acti: 


ality” of f roearch {terms Sweet 


ing. As we shall sec hollow, while some of the 
oniput indicators are fairly clearly tinked to scien: 
tific. production, their Finks with scientific | progress 
are_more complex and problematic. Vet it is with 


mnt concerned if we are to eva sate the extent to 


which scientists succeed in fulfilling the primary 
goal 2 hasic “research, the production of new 


4, Publications 


_ The first output measure: we shall consider is 
based on scientific piiblications. Several avihars 
have made use of the distinction between the 
“quantiiy” ~ = of research contributions 


heen eae in ‘erin of numbers of publica- 


tions, and has been the basis foc several studies of 
the growth af science, ° and of the performance of 
Broups of scicntists (c.g. Chang and Dieks [4}).* 


Ilawever, such studies have often eshibited a lack 


of conceptual clarity. as to what the number of 


publications actually measures. While it may be 


i Wisse dabsecan chiaiils oh cicieisiiaceee see 


important, poriicuarly-tn the initia! stages of dixccrminating 


as 


rogardid = a tcasiable nit of scientific 
ptexSuctinn, * its status as an-indicator of ats 


between numbers 0 a ™ rh. 
sis and the overall - “quality” or “merit” of 
the papers (eg. Clark isp: or the “eminence” of 
their authors (eg: Price (47, p. 41]). " However, ax 
we shall sce. helow, the.rclationship of these varia- 
bics to scientific progress isnot always straightfor- 
ward. Indeed. for some scientists, or groups of 
scientists, the correlation between “quantity” and 
“quality” “is. small or even zero (cf. Smith and 
Fiedler (50, p. 2284). 
The problem is that each publication Seek tt 


represent an equal contribution: to science. Some 


“mass-producers”. of publications make very little 
scientific progress, while other “perfectionists” 
achieve in a few publications very significant: sci- 
entific progress (cf. Cole and Cole (7. p. 382p. 
Various attempis have been made to circumvent 


this problem, for example, by “ weight ing” some 


publications diffcrently from others, * but this has 
tended to he done without any adequate theorct: ; 
ical basis for the choice of weights (cf. Bayes and 
Folger (2, p. 382]). Publication coants by them- 
selves [ail to “distinguish a the Muency 0 of 
genius and the loud noises.of empty vessels” 

ture [43}); they -give too each paras to the 
“operator” who produces quantity rather than qu- 
ality (cf, Bayer and: Folger (2. p: aed But how can 
One measure “quality"? = 


in scientific literature. Over_ is 
years. ago, Maddox (30; p. 1S] remarked that, “ 
%0 saad Naccid a literature it is difficult to tah 


Bai pecan = 
. Price atgues that “on the who there i; wheihier we like i. ¢ o 
Amt, 9 tenumebly pond correlation betweea the eminence of 


mew acientific kanwtedge, Uowever it has heen atniemed here 
(although this does weed empirical investigation) thar much 
of the infnrmotion pascing through these channch ends wp bn 
the Final deprsivery of aciemific pubsications. and hence thet 
only tesearch publications weed he esctessed (cf. Moravesit 
_ Wap. 
® Bee (tthert and Woolgar [20. Pp. 279-83} for « memmary of 


ranges, _ 
Lg Nemhers ef publications ate stea epperentty need by some 
administrator: tesponsible for allocating reanurces ia nctence. 


A recent article in Nenwe [45] refers to & profesee of 
chemistry (ond Nobel Laereste) being threatened with caput. 
ston by hit entversity for “inadequate preduciivliy"? 


scientict and his productivity of papers. It takes persisience 


and pereverance to he # good scientist! end these are fre: 


research... -icabise) sogiaieg tan fel of ucceoc C ca 
semee a ‘necessary’ cendition for the production of high 
quality work ...(and hecsute) the re 
such @ way es to encostage the © 
Productive and to divert the energies of the best creative 

— ectertisis inte athet channels”, 

° fee Bayer and Folger (2. p. 302} for 2 summary of sacl 
sttesmpts. 


{ weighting pi , 
the: ‘journal in which they appear (cz. Garfield 
(U*f. tMowever, there ix evidence that even the 
Prestigions. Journals. are _ net baal erie dis. 


have 6 been used ino obtain an —— of the 
of publications involve peer evaluation and 
bes * How these are linked el 


nit some allowance being made for varia 
” fi 


vide a measure of scientific pro- 
duction . is not a menture af scientific Prngress. tt 
“is what we shall term a partial indicator ™ of scten- 
tific progress - that is, a variahle determined partly 
hy-{a) “the level of Scientific progress made by the 


individual or gre 
nth 


tion (current and previous 
1S — see Crane {11.p. 703). the country, 
and the research area. Aonther factor is the en- 


” Giistion mabye tecsy Cal 0 arety at vies, porte 
tilerty_in the United States. where the National Scienc:- 
Foundation uses it to. esvese. for crampte, tts funding of 

” chemistry departments: where various eniversities employ i 
to decide cases af promotion and tcowre: and where citation 
evidence was produced. in-court-to prove (hat a, enman 
denied tenure was ns a ox_hetter than, two men 


a bndticnten’ 7” Hs teutohogi- 
cal in that _wse af the werd. cae sheady implies 8 
Pertiat_or Incomplete meawure. However, ta view of the 
tendency of seme users of Indicators tn Torget this perilatity 
on-incompletencss in the nature of indicatars, und the conse. 


quent danger of confusing the conceptual with the entologl- 
cal. Le. equating the menial contirnct with.“ reality”. tt 
weuth emphasizing the pertistity. Consequenity, the term 

“partial indicator” hes generatty been wsed threw shout the 


abbreviated tn ~hedicstor”, 


placed 0 on numbers of pulbfications for uh. 


Rd * The sittra- 


factots’ not only vary between scientists oF groups 
of scicatists, bul may afso change with time. '' 
Marcover, the question of whether (a) 


SIR: 
a set of entirely 
random influences whose-effects can he taken to 


cancel ont either for all large a aggrcyations of scien. 
tists or fos ao extended time: period. (Some. of the 


ily. In short, we 


can_conly.be-estnblished er 


have to consider why scientists publish papers, and 
to realise that seed do so not said So. present 


ay he cages to rely on 


pees of publications for assessing the scientific 
individual, or even of a small group, 
s that there may he wile. 


— Of (6) - For example, by selecting groups 
working in the. same specialty. under similar sys: 
= of organization aod fundi shlishi 


papers _in- the same j 
Soe then find. that the effects of (b). he to{a) 
can he reduced sufficiently to make_publication 
counts a» useful indicator of scientific 
Some pre‘iminary empirical results on this are 
presented in the latier part of this paper. 


ql mene papers 
he fees they count for, and the greater in 
eto publish rnove tcf. M TAL a . 
. eg Sutiivan et al. (55, pp. (52 84}, who find he 
Pindectivity of ecertale theoretical physicists hae quadrapted 
over 0 20-year perind, ovite that of experlmentabisis working 
in the some epeciahy has stayed comstont. 
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5. Ctiatians 


s scncisih: he td to 
2: “quality factor” inte the 


i papers, and that 
cach instance of such influence will-nianifest itself 
by the influenced paper. referring to the in- 


Tucaci 


lengeahie_ assumption, the 
c devotees cof citation analysis aS 


“worth” nf research work. pamaare kind ‘of evi- 
dence ix said to come from Nobel Laureates: the 
majority of whom arc‘amongst the top 0.1% maxi 


cited authors (cf. Garfield UK 4 p. 485), giving tise 
ta the SUaRi 
c 5. (Garfield {16. p. 671). On 


and other evidence, "it has heen 
t “the best 
cator of the warth of research” (Porter 


fow justified i is this claim? 


ions face a number. of pate 
same: (echnical, some. more substantive (cf. Cole 
and Colc [9, 
Science Citation Index... which is 


generally oe in these studies. does not provide a 


totally accurate reflection of citation structure (cf. 
Moravesik yen : For example, : Sitations to multi- 


regardless of whether-they were the first-named 


author), it Is gencrafly assumed that using only 


** tna ter paper, Cote and Cole 19. oP: 29- 4p make © 

ton hetween.“ahsotute quality” (1c. Papers embndy- 
ing “ahsotate truth) and “soctalty determined quality”, but 
they ett maintain thet citations “are se edequale measure 

- -- of the quality of work socially deflned™. 

* See eg Rayer and Folger {2}; Cole and Coe |7}; Myers 41). 


67 


Cole and Cole {9. pp. 32 
true on average. for. ment 


groups of -scientists who engage in 


grees: nf collaboration in-their research; use of 

first-named author citations may lead tn signifi- 
cant systematic errors, and-in the empirical. work 
descrihed below total citations are used Tor this 


reason. a secnnd Acchnical 


} use one i tial and some: 
times two, or allernatively because of a change of 
name on marrying; these, however, can generally 
he checked. where such varialions. ate suspecied. 
Similarly. some names prove “difficult” for « citing 
authors to spell and a have t ‘ 
Un cf several different Andex. 
Yet annther problem at-of two authors with 
the same 5 name and i initials. hut this can gencrally 
ided the research interests of. the 


described. jee. ‘one difficulty concerned. p 
which are: listed together in the. Se < 


cases 2 Specapabeatad 3 papers it was not ot possible 10 


in our. — in 


. and journal- names. 


d accurate list of publica- 
al concerned. In short, most 
of the technical errors can be overcome with cate 
and diligence; and this effort would scem well 
warranted if Porter's (46, pp. 264] estimate thai, if 
tyncnrrected, these crrors could be as large as 25%, 
Ik anywhere near accurate. 

The substantive problems associated with ci- 


tatlon analysis arc: more complex and severe. First, 
case of high-quality work that is-ini- 
tially resisted or neglected, perhaps because it is 


“ahead of its time", and which therefore accy- 


mutates few citations, The assertion that such oc: 
Cutrences _aré raré (el. Cole and Cole, [9 PP. 


FC that these eccur most 
isteneatl. Secondly, poor quality papers may 


he frequently cited eck they are cither con- 


cn” (cf. cg. Janke [23, p. 


#92). The response. of Cole and Cole [10. 1 pn. 
32-33] tn this criticiem is that, 


Much important work turns out in his orial 


__ While there is more than a fittle validity to this 
argument, it iMustrates -the mauler contusion 


rc BR. Martin ond J Irnine J Aviesiing burtie veqenrsh 


Sie: TA anor if one is attempt. 
ing tn compare research in different specialties: or 
to compare research i in the same specialty at differ. 


ent times: = aes 


degree of “openness” of the individual of Erowp : 
concerned to work carricd oul elsewhere. Mathe- 
son’ 's (33. p. 208} stanly of Drilish university chem. 


can vary considerably (by a factor-of (wo) between 


differen’ research--groups working in the: same 


ficld, and this-suggests that, at feast for groups 


8 hy 
tal ideas, Once aa ideas have been accepted and 
bully ee into the body of scientific know- 
rs_may che 


practicing a high degree of sclf-citation, the aum- 
her. of Eitations received (may | be at feast partially 


is not 
Papers or with scientists. Sse high-quality 
paper in a small, narrow, or highty specialized 
field may produce few citations - fewer thao a 
Similar quislily paper in a more popular field (cl. 
McGervey. [35..p. 30}). On the basis of compari- 
sons between solid-state physics and the smaller 
specialty - of high-energy- physics (although - this 
would appear _to-be-a far from typical “small” 
field), Cole and Cole (9, PP. 28-29} conclude there 
is 1-0 evidence for such an effect. though they do 
admit that it might he a problem fot very small 
fields. ‘ Other authors dispate this claim by the 


The Cokes jt, Pr 32-33) comtinwe: “Why shold « barge 


member of_scientias waste thels deme pointing out trivial 


fecelve cttical citations 71 wot accumulate large members 


- od citations” - te Su Leet 
” Citation patierne within a speclahy do change considerably 


are more ayaa cited than others of similar 
quality: For example, authors in theoretical physics 


Per paper fomped free 7.1 tn 9.9 ducing the First_yenr of 
fesearch inthe mew and eapitty growing field of pulse 
research. Similarity, Solfiven et al. 151] note a dowbbing In the 
averege wember of references per-paper “over @ 20-year 
Perind in the rescorch arce of wenk bnteraciions, 2 subfield 
ot high-energy physics. {n comnast, it te atgwed by some (eg. 
Lewani [28, p. JO)-that the size af the research field will net 
Brextly infisence the relationship helecen the quelity-of-2 
Publicstion and the nember of ciatinas H eeceives hecawse. 


quality 1. o broader -_fictd, However, there must he dewhts as 
popper tbsshtrds cee Wrasbrahioleniy hace 


the peicouinge of sel cations dorgged trem 52 of the 
total cleations tn 10% over the fire year nf pulser research os 


with the slzeol the specialty. For example, Mesdews and 
Conace 1341 thew that the average Gember of relercnces 
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more. potcatiqity citable work by other authors became 
avateble, 
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nology. Papers 
" 3 xiandard technique 'S may he highly 
Cited _repardiess of whether or not the. fest, pub. 
lished dexcription of the téchniqne cnastiiutes 3 
high-quality piece of research (cf. Nature (4. p. 
699)). Hence, the number of Citations, a- paper 


receives may reflect its mature as well as its quality. 


- Finally, the use of citation counts: is. com- 
plicated hy the “halo effect” —_ the fact that: when 
there is a. choice of publications to Gite, the auilior 
Of a paper-is mare likely to refer to the work of 
eminent scientists (cf. Smith. and _Fiedler {50. ?. 


229)). Scientists who have gained a reputation hy 


publishing significant research in the -pasi then 


enjoy a halo effect: in that_all of their. sesearch: 
even if srene of it_is of lower quality. gains addi- 
sional attention. Cole and. Cole Lig! admit that, i 


Unileriving ali these prohicms with the usc of 


Citations as 2 measure of quality is our ignorance 
of the reasons why: authors, cule particular pisces af 


giving. in which citaticns are held to reflect prim- 


arity scientific appreciation of previous work of 
high quality or importance, and potcntial citers all 


have the same chance to cite Particular papers 


(regardless of who the authors arc, where they 


publishod. what language they wrote in, etc. - 


other words a “(ree market” in scientific ideas). 
The model also assumes that the norms of citation 
behaviour are largely unaffected by any “external” ; 
Pressures: (including the awsrencss of scienii 
Skat Gitation analyses may be used to assess their 

scientific performance), and that all citations are — 


of equal valve or inte ft. Such a model ix obviously 


research and that this docs nor therefore affect the 
une of citations as a measure of quality. There are, 


argument. First, the quality of currently cited work 
May not be ns good ss that of the previous re- 
search that determines the size of the halo effect 
(ef. Croom 112. -p. -VI73p: Secondly, the- refa- 
fionship between the halo effect and the quality of 
fast research may not be « Hnear one, with the 


leading representation of reference-giving:, the ae: 
sivit of referente- “Biving is, ix. like aiher aspecis a 


tutional position,and social and Sah tical goals, 
Before we fonk to see how this can be done in 
Practice. certain conceptual distinctions must be 
made in order to ascertain what the numbers of 
Giations may or niay not indicate. 


result that. analysis based on numbers of citations 
could exaggerate the differences hetween high qu- 
stity and low quality research. 7° For czample, the 
halo effect enjoyed by 4 Nobel Prize: winner -is 
probably. far largest: than. that accorded to the 
winner of a British Royal Socicty medal (often for 


Cinrity as ‘to wi at the citation ratex measure. To 


work of not greatly inferior quality) because of the 


tremendous publicity given by the scmi-scientific 
* O.1T. Lowry, the nat highly chied achentist, rexelarty re 
ceives several thewwnd chations each year (seversl_ times 
fmnre than even the-man eminent of sctentisis), the fut of 
which refer ins technique for proteia. determination: Given 


that this techolque le neve 60 siendard, ane wonders eheiher 
ah thee whe che this wnrk heave in tect read the relevani 
paper belore they mokewse oft, 

Sd This may partly oconeet tne the controversial conclwsian af 
Cole ond Cole Jf, p. 368} that “pnly @ lew achentisis cnmisth- 
te to ecieniifec pengresa™. 


avercome these problems, some authors “have at- 


tempted to distinguish between the “quality” and 
the * “impact” "of publications, arguing 


naper, they do at least reflect 


per, | at le siaeael oath 
scientific commnnity (e.g. Garfield (15, p. 0p. 


n Cl. tnbistier and Pzdnowck itt. P. 33}: “tn practice, the 
serl-sctentific and_papwlat press coneiderebly diuinrt. ihe 
vielt ‘ty of the different fneme of recognition. This may 
Produce @ gop between the perceived and true quality of 
ectentific work.” 


Pa a 
re 
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i 1s percept which spec eal will 
rates measure (cf. Lawani (2%. p. 279). The- first of the most fundamental” questions (Cole and Cole 


these concepts refers tn the - posers itself, while (10, p. 32), which find it casiest_tn attract sesearch 


jer Iwo arc more external, referring In the funding. which are more Likely tn attract high 
s beiween the research and other research quality —— -and. S0 on. - in nilter words, 
arcas, and describing the strength of the finks tn, 


or me the inplix tions. for. other. vescaich ‘activities, = 


hetically picasing the paper in_a dormant or declining one. ??40 sar we 
mutations are, how original | the can see that the “quality” of 3 publication. need 
re, and so on. But quality is stilt not he synonymous with its contribution tn scien- 
ve rather than absolute, and it is socially: as. sifie knowledge; for cxaniple. a high-quality Paper 
well as cognitively determined: it is not just intrix-- 7 


sic in the research, hut is aomch judged hy 


tion. of -the- size of its Saadiioe important 


thors express them- 
th in a journal with a” 


paper. f:ven = same indiviiea! Vimay evalu 


tn | knowle dee. at 
thot time (abou its impact may of course, alter 


would have if there were 


perfect communication in science (in short, if there 
were the completely “free market” of aenial 
ideas ‘mentinned above). Mowever, there. are” 

perfections” in the ‘scientific communications psa 
tem: the result of which is that the (mportance ofa 


papet may not be identical with ‘Als tenport 


sit possible tn_obtain any. absolute or direct 
measure of — importance, ONG The ofa 


tors_xs the focation of -¢ 


prestige, language, and avaitahifity. of the : publish. published in Wen Altice ore cited less than those pubfithed 


ier Journal ch. Dicks. and Chang (03. p. 247p. 


eversens._ —_ 
iret Concepts, we * CL-SMe Eb DO oho Rade vont its tH 


t next examinc ‘how they are rclated tn cach dian mai ic ca Po 


40 the notlon of eclentific progress him. Parity i may be de to the bniinaic quality ef the 
described earlier. The importance and the Impact tapers ond partly te the high vthitity of ihe man and bis 
work” 


of a publication are determined hy the relations 
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f 2 c ev: af ifirectly, bat only 
appichcarled indirectly there the perceptins of 


irect reflection 


citation Practices) re scientists, The- number af 


(a “measnre™”) of its. quality or importance. nor 
even of its impact. The citation rate is_a poetical 
indicator. cial She e impact of scienti cier tific  Paiblication- 


knowioete: ‘but also ite’ by (h) =e fac: 


cxample._ the reading. and. referencing habits af 
different individuals i in different institutions. coun. 


ene: Oe aa waka and their ¢ ae 
institution. As with numbers of publications, it 
cannot. be. assumed that. the effectsof (b). are 


ificant compared to those of (ay. 2 big 


ton ies may be andor, ¢ 


ing different cognitive and social Sees 


telative importance of (9) and (b) on Citation rates 
can only be established empirically - for cxample 


4 Kapton (24, p,183] warned thatthe ene of citation ausiysie 
a8 9 tnd fer evalusting science covid tead to changes in 
Chalion practices, and in the enutie of eur interviews with 


" 
hy looking to sce whether there are systematic 
variations in seferencing, | behaviour herween iiffer- 
ent gremps ¢ ientists? 


the methodological prohiems associated with cita- 

tians that have heen detailed above arc consider. 

o diminished. First, an ‘important eaipecian | 
le Cited. ir y 


WW 
sis per se-(ef. Lawani (28, p. 29). Indeed the: use 


nf citation rates ta identify those pioneering papers 

whose importance is not immediaicly recngnised 

may well bean essential first step.in ascertaining 

whi recognition ow their importance is not | always 
y Tort : sial, 


further research aimed at oad. upon, oF te: 
futing 


lates and therefore contributes to scientific: pro- 


gress. Thirdly, if citation rates are taken to reflect 
the impsct_of a-paper rather than its quality: or 
importance, the fact that, in cases where there lias 
heen complete integration of ceriain basic idess, 


the original publications currently receive few, if 


a citations, can land ineerpeetod as s implying chat 


3" Kaplan (24, pp. (82- =]: einer » some: of the questions thal 


Ad] 
aceume thal: the: Incidence of such effects by care and their 
magniione incignificent, thes enshling them to claim tha 
"chatinne_ general, representa avthentic indicate of in- 
Recnre™. (Ut should he nied that this represents a weaken: 


fon from thels esrtice clin thet the number of ciatlens 
represents the quality of a paper - ece Cole ond Cole [7. p. 


17h) 


incon reperted?...De citation practices relies significant 
elements of the nermative and valve sysiems of ecieniinttT” 
Thee questions have largely gone wnantwered siece ihen. 
Bithowght sume recent work on the classification of citatines 


ny eventually provide answers in such quazilons, 
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now For- exant le, F: pee ns 1905 prpers are still 
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hecause: har y anyone cls reas the papers. 
Helore- they r reached - stich a. stage of inte 


witts ather contemporary paldicatons. Since that 
time the impact of these paper: has diminished, 
while “the insportance of the ideas. contained in 
them has remained high and has been transferred 


and saccetsive b esipaala 


ation- of parca | tare ‘ant the “grandparent” aad 


preven | antecedent papers (cf. Ghee cos BI, Pp. 


Pact" provided that we recognise that this is an 


synon synonymous with “importance”. a | quality which 
of papers _in such away that citations do not 
reveal the “founding, father” “paper, 

. The fourth protiiem, that_of the higiaaatiy 
paper i in a small unpopular ficld which gains less 
citations than these in large active fields, has al- 
ready ‘been dealt with - the lower number -of 


must -he rcengnised that 
in the case of scif-citations, the : re sfiect of 
fh). the “other factors”, on the Citation rate may 


very_well be greater | than | ia the case of “normal” 
citations: 


empi tly the. effect.at inchoding hath self-cita- 
tions sind-in-bouse citations (citations te the work 
of colleagues: within the centre) in order to sce 
whether the effect is significant or not, and this is 


attempted in the empirical study described besow. 


7. Peer evaluation 

__ The method of assessing contributions to scien- 
tific knowledge apparently. mast favoured by sci: 
Cotixts ix that. of pecr. evaluation. ** , 


evaluation does. not yield a simple measure of 


Gitations being interpreted as representing a smaller 
impact on the advance of scieutific knowledge (in 
the sense af having implications for the work of a 
amatler numberof researchers). Similarty, the fect 
th at certain kinds of fesca search. Papers. ers (or papers 


ocive more citations than average can be interpre- 
ted as indicating that such papers, whatever their 
quality, do create. a wider_impact. than average. 
For example, « paper describing a minor improve- 
ment -to an extablished method may constituie a 
telatively small or low-level contribution to scien- 


tific _ knowledge, but when integrated with other 


contributions +t can have a big ixapact_on scientific 
activities (and hence on subsequent scientific pro- 
grexx), alicring the procedure Involved_in some 
standard and widely used technique; and this wifl 


prohably be reflected in itx citations, al least until 


Mets, integrated intn the body of aclentific know!- 
edge. 7" “ 


** There is, hemever, a profes with clotinne in thet srene 
tondard techniques, even after having become folly 
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subjective nature of this method: peer evaluation 
is based on individual: scientists’ perception: of 
contributions by others to screntifie progress. per- 
Cepticins attived at through a Complicated scrics of 


tatellectual and social processes, mediated try -fac- 


tors nther than the quality, importance, or impact 


secepted, ill do peneatectstions in che original paper os 


tn. the. pertcin-determinstina teciiniques.rclerted.in. in ite 
(9, while others (for_crample, aperiere-symthesis ia 1 din 
Sriranceny) 6a arnt. The teenie for such diflererr 2 in 
ciation patierne need further eopirical investigation, = 
7 Cut of -the 69 scientints inerotewed in-this stedy, 76% 0: > 
peer evaluation os the best. ncthod..and 19S caw. at equal 
ben with either publication cruats er ciation enalysis. 
Fairty steniler ceaytis ore civelned by Chen $3] bn bis study 
of university fecutty staf! ead admintzerators ow the retative 


errnt_valid indicators, although i ts nat etanluicly chee: bn 
this: stedy whether the latter refers merely tn mancher: of 


[reblications, or whether i wes interpreted hy the seapem- 
denis in the-swrvey on incteding some assers ment o phe 
quality er infnence of thre papera ga well. 
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jal indicate af cientific 


ace: it as anather p: 
progress. 

Fes: and most: importantly, we ment fecognise 
ihe effect: nf pol and sncial pressures within 
ty on the way ceriain s 
tists wall aliate the worth. of their peers, _Moitern 


prestige - cand ing. Research groups, particn- 
farly those in areas ©! science characterized by 
capid scientific growth, camplete fiercely for the 
rece and acceptance of their ideas and the. 


arsaciated with this rivalry. Scientists w 


asked to evaluate rescarch may -he influenced by 
(5 _gassihle: implications of the peer evaluation 

th for thee own (-10res a-d lor these of their 
caillcagues. OF course, cine cold try to rediic. she 
cHloris. nf this Brvles, by asking every peer cen 


tion and. ic aggregating or averaging the re: 
sponses: bot giver that this ‘s seldom. practical: it 


is gene ally onty possible -to as 


‘tempt to 
tives from all the importent groups in the ficld are 
involved in the assessment in order to minimise 


these probisms, 
Representative sampling also helps red 
effects of 2 secon? croblem with peer evaluation, 


namely that_n? altowing 1 for the diversity in the 


cognitive tocatisis_of it~: reviewers. °° There_ is 
inevitably a tendency. for scientists to_evaluate the 
worth Ls, & scientific fontribulion:i in terms of their 


“callce-reom” discussions oF On the reputation of 
scicatists (in particular, thelr ceputation for peri: 


ming at conferences) than on their actual enniri- 


eminent scientists with a wide knowicegpe af the 
fiekl concerned: hut this can introduce other prob. 
lems (sucli as that of caching ihe views wi less 


ae with all attempts to gauge opinion. - the 
Propensity of people in conform in conventingalty 
accepted (patterns “of _ heel” Although | perhaps 
privaicly holding the work of certain other well- 
ki J 


pre 
their x.sessments arc to he treated confidentialty). 
Such conformis: bchaviour need now_always he the 


tke “hain effect” by vAhich work may be evaluated 


more highi: necaus: of its association with a suc- 


more, ipusiicolaily § in ces cases where evaluators 
de_not possess afl the knowledge needed to‘make a 
nappa jirdgencnt, yet do nat want (6 withdraw 


accorded ta work, for example in the form of 
Prizes or medals, rather than on the work itself. 
There is then the additional problem: of taking 
the time-lag between the making of the 


hetion to science, and its subsequent recog- 


nition. —__ 
Because of these difficulties. peer evaluation 
cannot claim to provide an incontrovertible mea- 


sure of scientific progress. As with the other niea- 


sures we have considered, it isat best a partial 


Muenced partly by the size of the contribution to 


= * tence, ie 


serch a3 publications ‘end citation: The termer ts: linked t0 


the inttes ln thet peer evoleatina is bored partly an journal 
firersinre, and sree aepects Of journal Uterature (for cna 


* CO. the dicewsston by Mitrolt and Clanbin (15. p. 199] of the 
weed 17 cremine the eng vies of reviewers and re- 
wteeces in order tn mar sence of the vetulis af peer review, 


where peer evaluation ts combined with other indicators of 


actentific progress, the veriovs methads of evalustion ore 
Tound to coincide fairty weil fel. thd }. 
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progress and partly by ether Factors. the 


relative importance of which cannot be assumed to 


be insignificant. 


formal Sa afforded to scientists by the 


scientists through the awarding of various honours 
such--as_ medals, prizes, invitations | 
major_conferences. of election to a cational 
acadeiny of science. The allocation of such recog: 
Nition involves a prior process of pecr cvalsation: 
and is, as sich, based act only on scientists’ pcr- 
ceplions_nf conirihutions (o knowledge, biti also 
On numerous other factors, the clfects of which 
may vary considerably. For cxzample, Cranc [11. p. 
740] claims to Find-that affiiation tn a prestigious 


university t% more fikely to bead to recognition for 


Scientific community. In the cate of “discoveries”, 
the crucial questions centreon who should identify 
the “discoveries”: and using whet criteria. The vse 
of the notion of “discoveries” therefore involves 
— ‘sienilar to- those encountered with Pocr 


science for identi yin Scone See oe mrad 


judging which contributions to knowledge are sul- 
ficiently influential to merit mention es “dis- 
ceverics”, and” which are not. Moreover, such 
jiulgements_ reflect (preoniling perceptions of in- 
Nlucnc=, which can be substantially differen 
those held at the time the particula contribution 


a scientist than high productivity. Such a result 
leads to doubts about the adequacy of recognition 
as even a partial indicator of contrihutions_to 
scientific Progress, tinless preceded by a thorough - 
analysis of the social sti of the scientific © 
ty and of its mechanisms for allocating 


A wnethodolegy for accessing contrilestions to 
ees knontedge - 


in “preceding. sections we liave seen. thaw: all 
quantitative “measures of scicnce™ are at best 
partial indicators, influcnced by a nciwork of 
ioterrelated factors. of which the size of contribu- 


tion (0 scientific progress és butt one. Nevertheless, 


to iia tebe was madc. A contribution that-is 


given that all: decisions on the allocation of 


ly highly § influential may subsequently © come 


reconstructions found in “official histories”; even 
though: it may have stimulated: much important 
tesearch at the ragdoll One “might use 


resources between different areas of science must 
involve some assessment of the respective merits of 
the: various arcas, oe would argue that. sclective 
and careful use of these indicators (despite the 
associated. methodological. and conceptual prob- 
tems) is far better than nonc at all (cf. Moravesik 


(38). Previous use of these indicators, -in-the field 


that has come to be-known as “scientometrics”, 


valve looking at_the- references cited in: review 
articles, thus bringing one face-to-face with the 
Problems of citations mentioned earlier. tt is true 
thal the citation practices in review articles may be 
less waywred or self-centred. and less subject to 
the effects of the “other factors” described above, 
than citation practices In research publications 


(perhaps -because. a -more comprehensive 


fiterature-search generally precedes the writing of 


a _zeview, or because only the “hetter™ scientists 


are chosen to perform this task). Ulowever, this can 
only. be investigated, empirically - - H cannot be 
esutmed. - 

"The oiher Indicator of sclentinic progress we 
should consider Is the recognition accorded to 
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has_met with a barrage of criticism, in particular 
from scientists. The purpose of the previous sec- 
tions has been to show why much of this criticism 
is — For example, assexsments of tri 


ions can 
fess when little or ne 


that Indicators_of science have in the past been 
applicd In a_ simplistic o¢ irresponsihle manner 
doesnot mean “that they cannot (be usefully 
employed at aff. On the conirary, having analysed 
the shoricomings and conceptual difficulties axso- 


elated with each of them, we arc now in a better 


position to see how they might be used to achieve 


. 
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aoe the sbacecs of any ape oe single mca- 
Tr of scieniific progress. the only way farward 
pear to be through the combination of 


tial indicator is influenced to a greater on lesser 


extcat hy numerous other actors i in addition tc to 


volved, one must attempt to “conical for: these 
othe: factors hy chonsing the groups of scientists 
tp he compared in such a_way that one can cx- 
amine. the size of the: effects of the “other factors” 


be assumed that the 


5 Parison, 
what i is ideally reepscred is thal there should. he two 
OF note groups, working in the same specia 
a a similar time: spetiod. publishing # in: the 


resmurces, and situated in = si 


context. _For example, a high-energy physics group 
R 


berford Laboratory might 
for Daresbury Laboratory 
(and bath with similar centres, overseas), Similarly. 


has been kept small (ie. the matching of the 


groups has been-largely successful). and ¢ 
indicators therefore provide a reasonable estimate 
of the contribution to scientific progress made hy 


different fexcarch Sroups.. The. extent, fo: which 


* the empirical siudy described here. 
- As mentioned before, the study focusses not on 
r rch 


it 
several of the problems aia ‘indicators discussed 
earlicr ace diminished. In. particular. the influence 
0G the indicators of the ‘size of the contribution to 
scientific knowledge is less likely to be swamped 


factors” whose effects are systematic. Fig. 2 sum- 
marizes the main problems with the varicus indi- 
cators_and how we have attempted to minimize 
their effects in our ney: 


geepe 
tnchede Westtrant 48 (ovebl 126.27), Matheson (99 and 
Andrews pip 
"tn the cane of lndividwats, lH ty walitely that one will find 


rather than individuals, however, no“ matching” is 


perfect. Thus, while Cambridge and Jodrell! Bank 
are hoth engaged in radio astronomy, they tend to 
concenirate om different research areas and pied 


equipment - cat each centre: . Calanige. has con- 


iry), while Jodreli Bank has pla 
emphasis an “hig dishes” (used cither individually. 
of together for bong-baseline interferometry). It_is 
therefore cssential to include other centres in this 
stody to ascertain whether this divergence.in em- 
phasis leads to systematic differences i in the effects 
on the partial indicators. For thi 
Netherlands F< 


ml 
ferometes and the receiver at Dwingcloo, and the 
Max-Planck-Institut_(dr_Radioastronomie (MPI) 
group based on the Bonn “big dish”. were 


chosen.>* These latter two centres are essentially 


raeinies at facilities (as Opposed to t the ==! 


counterparts. Nevertheless, riven the simi- 


larities in the size and scale of activity of the 


” inevhahiy. ee ere ert The 


Ronn big dich iz more precise than the one at Jnirell Rank, 
ond ts iherelore used foe work at shorier wavelengths and 
fon me specttencopy. ta the cate of the Dutch and the two 


more-than @ very email aumber whose {fairly aavrne) ve- 
search interests entacide or even overtop remonably com- 
pletely; wth: groups, however, a wider sange ol research 
teterests Us covered and ht chowid he presthie tn find several 


foe which there it @ sabstantiol degree of cveriap. 


main Cambridge inierferomerers, the latter are vad prim- 


avily foe continewra work, while the fosmer carries can both 
conilnwum and speciral-line work. Nonetheless, ihe degree 
of comporshilty between the centres, while not compleie, fa 
sufficient fre the enstysis thai follows. 
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« — Use ony! indicators that yield convergent/semulté | | 


ral Pees = tr) Perceived lonptication of - ya) Use a enenplcte amide, o 


evaluation resets far own cowirc and 8 large representative. 
sample (25% or more) 
a) Ux verbal rather itan i 
scientific ennitibetions in written snrvey pn can peeas ‘ 
Felotion to thelr con (very evelnaine if 8 divergence t 
Giflerent) engnitive ond hetwern expressed opini. ; 
ancial tacationn. - emt sad actual views i: sys. : 
eaperee§ epee < pried - - 
(9) “Coudormict” aaeeasments OQ) Asmre evaluators of confi- 
{e.g. “halo effect") socen- _. Heatialny 


() Check, few systematic 


HIg 2. Moin canes with the varies partin’ Indicators of actentific progress and detelis of hw their etfects may be minimired 


_thelr cpenaen as whether carryl nal research nn-an. iene 
national frcilitles, and their use of the same Jour- cather than a big dish leads 10 systematic dif 


nals for publication; it showld be possible tn see ferences in numbers of publicatinns of cilation 
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rates, If ihere are systeniatic differences. anil 
if. they are consistent with the rexulis hased on 


than mmcrely being. a consequence of 


ly 

sed for comparisons between Cambridge and 
Jodrell Rank; it is notappropriate for comparisons 
helwecn centres ia ‘different countries. ‘There. are 


“discnvery”. The number of major discoverics ix 
rather sriall for this to he used in a systematic 
y. while broadening the categor 


adopting highly cited papers as an zrdicatnr. 


to. Tinpat mencines 


Before considering each of thie partial indicates 
of scientific progress, we musi first fook at the 


scale of scientific activity at ench centre: tere are 


ba Tor caample,. i ts Goubiful whether “one gan equate s 


- Jefech = NFRA MP 
hedge = Wank 


7 3s -4 - 
6 ‘4 -0  -55 
3 39 ~30 ~5 


ES STSLSs Sst 
_ astronomical research. - mae hee ia 
* This exctudes MSc. stedtents y : 


anprecialie diffcrences_in the sizes of the four 
Broups. and i -woutld therefore he. misleading. lo 


most important. being the. numberof scientists 
actually engaged. in: research. The. 1978 figures ?* 
for this are set out in table §-: an hat, 


in terms of resea 


Fellowship of the Royal Sockets: with membership 
of the National Academy of Sciences in the United States 
hecawse of diflerences in the percentages of prociising scien. 
tiste selected in each cnwniry, ic the criterias used in sefec- 


tine, and $0 on. 


$1 ts probehty significant, however, that the Cambridge 
Brown, . ot er, appears So have contin. 

tated most ta sclentific progress, factudes two wenticts who 
shared the Nobel Prite for Physica ln (974, 


of the groups. There to. for example. 6 Wigh degree of 
concensus that the first ohvervations of pulsars represented 8 
majoe discovery, bet far less comsenses mer «tines lower. 
tevel achiewements. 


in group over 
research students are included, however, these dit. 


9 (ently, the corresponding figures for earlier “years ‘should 
atso be inchided, but these have proved far harder to otiin. 


enced_by_the® Ich: groups over the ten ‘years. Hence: 
during there last five years, the rasies of the ncate of activity 


ot the four ceaires have ant changed vey significantly. 
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on WR Morr and J, Tevene 7 Aucrting have erwarch 


ts (lection number of netrnmeeny cexeatt hers in 1978 afer alle. 
ang, low teaching duties ® 7 


Com. Jecireft NERA (4Pt 
—— eitge _ lank 


Braduate and M.Sc courses, while Dutch and par- 


ticularly German astronomers have far Fighter 


tcaching tnads. Clearly the more teaching Abat is 
carricd ont, the fess time that remains for research, 
and_one_way af considering this is as 3 reduction 


in: the effective number of asitanomy resea 


he scen that this accen 


scale of research between the two British Cenires 
and their swn continental Counterparis. * 
A third actor to ed considered és the annual 


‘cup. The 1978 values for 


these are presented | apa 3, together with vari- 
ous figures on the costs per researcher depending 
an whether students are included or not, and on 


whether dilferences in teaching duties ate allawed 


. for, The figures on total running costs show simi- 
“far a larger differences in-scale than the 


Figu he 

being qiaesiae: tn the higher salaries in Holland 
and Germany (sce the ‘figures. on onsis. per re- 
searcher). In addition, the major capital “er 


* Astnnemers fea ads aotein pal hee 
vescarch-related adminisicatinn both within their centre and 
entaide (on SS Sens eens rei fe 


acs ecparanel bes Gt davcead ba sesiarck lar dig seioos 
that these.deties sre om emential end integral port of mad: 
ern, loege-scate revemrch. While research centres overall spend 
“a toughly similer traction of thels time on lnsiremental texks 
and administration, this tesctinn obvinusty varies cychically 


#1 a centre parses thremph the stages of deshgning © major 


tescarch facility, constrection, captoitation, devigning & ow 


facility, and son. ic gesruah ae apace ead 


Hectusie appeecistty from ene year to the neat depending 
on whether a major wew piece of cquipmen! le being bulls, in 
calculating the running cose tor cach cenire, we have_ex- 
choded major Herm of capital expenditure (of greater then 
£100,000). Ulowever, ‘ronning conte’. as defined here, do 
cower afl other expenditure, Incinding the erat of purchasing 
fen etpencive ond more routine liems af equipment, 


ta ” -p - 


WE OMS -10% ~ 108 


on teaching wndlereodrate. MSc. 

_ Droch ~Dactorsnden”, sed Crman. Diplinma stedenta, 

* Individual asiramemers were athes! hve emech af thew time 
wan dromed & teaching. sad the rrauh« were averaged fow 


each centre tn give thene figures 


ment-costs (that is, the cost of a new telexcone, or 
of extensive refu xisti 

ment) are summarized in. table 4, Ji thould he 
noted that, if these are. spread over the likely 
lifetime of &. radio tclesenpe. (20 10 years. | for cxam- 


ning costs, even after allowing for the effects of 
inflation. Consequently, it is the canning costs that 
largely determine the rating of the total costs for 
the eaail : 


support st ‘ae non-astranomers), and the fig: 
ures for this are shown in the second tow oa 


table 5, along with valucs for the ratio of s 


group Is nearly three times better Off (a thls re: 


spect. However, the: catlos of kupport stall to. as- 


= icators. suggest 
that the scale of scientific activity is greatest for 
the German group, which is well over twice a» 


large as the smallest group. at Cambridge Jocirett 
‘ 


RR Martin and J treme | Anvcuing futur research ” 


Cable 1 - 
Anenal conning cots foe 1978" . a § eating) 


135% £35k (65h CAoh 


Indeett = Mark $250.11 
Bank Mark $9 125 x 81-10 tele- eet pane 


196-4 o3mM 
1967 01M 
1974 103M 
1974 £0,724 
_ Kennchin N telesenpe 1977 f9M 


however, aaa isBiseai fee om dares Bede we mea 
that in an cashier drat of this paper. e rather wider definition 


=. 


{the salaries for eppresimately. 3 
tenearchers and stedents (and of 


regularty wse the MP1 telescope, __ 
* te after exchoding time spewt teaching. 


ling popularised accounts of scien: 


pe 
Dank appears to be about 20% Uaiger than Cam- tific research, unpublished work; preprints, “? and 


bridge, and the Dutch group over SO% larger: tn 


terms.of running costs, the differences in scale 


between the groups are apparently- more _pro- “* These gure ca are haved oa the cong fete publication fists 
now 


inced, - probably 4 ‘because of the higher salaries provided for us by each of the centres, and ihey should 
in-continental Europe, with the Dutch group cost- Ss we renmomoliiy comes oe there ore errors, they are 
ing about three times as much and the German a 7 tele more tan 2 
eo oe fur tet mock at ech he anon io ney 
British groups. tie-crmniing”. 


pe | 
2) 
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Neer of sippenrt aialt im 1978" 
__bripe Rank 


Member of sopperwt 


© Ache the previous taller, the figuees may he lightly bo inscew- 
cate became af the dilficwhies in  eathmaring what to. incr ” * 


other cescarch groupe, For exavnpde, the stall of the Mullard 
Radia Actronreny Orervatery at Comnbriige ts 04, hut there 


20 19 23 Pa . 


134 


a) BR. Marwa aad J levee 7 Airing hewe cevenerh 


Table & 
Nembers af pabhestnene 


Com. Jeske NTRA Ard 


ate in edditinn same employees, such a clreners, 


telephnse 
settchhnasd aperaters, ond thrariam who are -shere! oth 


University, 
the total 10 abrat $0. The overall accuracy of the figures in 
the table should, however, he sufficiew! few eur purpemes. _ 
© This inchedes 30 eniversity ‘sstronamers ond 10 students ot 
Leiden, Groningen, and Utrecht. sho carry owt their research 


Primarity on the NFRA [ocitiies, together with an.estlenated 
tS muppet siatt (secretaries, cleaners, cic.) at iheve eniverst- 
tes, a 

© ‘Thi inchedes 15 axironomers and stedlents at Bone, Ulam: 
tong. Rechum, and-Tahingen, together with on estimated 6 
- support stall at-theee wriversiticn, - . - - 

* ‘This cachodes 16 siall Jormalty employed 40 qwovida services 
tn Jodrell Nank_ vitor: (ia the Concourse Ruliding. re- 
* ptawewnt, and Arhrrcium), « member of whom ano provide 
Some seppont services bp the raain atmonanener 


taboratg reports. are all excluded. 
:From these figiires. one can nee that the fevel of 
iraduction of the Cambridge group grew 
untit the fate 1970s when it levelled 
( “dip™ in. ovtput in the carly 1970s 


coincides with the-peried during which a large 
proportion of the_group’s effort. was devoted: to 
ringing the:new $-km Interferometer Into opers- 
tliin (sce talite 4): In the ease of Jodrell Bank, the 
fevel of production grew untll the end of the 1950s, 
during which time It was consistently higher than 


hedge enh 
Publications en - : 
1946 ' ' 
1947 1 3 
19an 6 “ 
fea 2 3 
1930 3 a 
198 6 iB 
1952 9 14 
1053 s ty 
i984 i] bi) 
1933 y mm 
19596 10 "9 
1957 m cal 
1988 10 7 
tose 6 pC] 
tos Bw zu 
1968 2 Fi) 
1962 "9 zm 
1943 rh] n 
tone 12 bs 
1963 a rh] 
1966 » t 
1947 4a "5 
1968 an 22 1s 
968 ] it té 6 
ten HN 2 22 4s 
tyme 2 2 a a 
1972 4) ” 7 3s 
973 ” 2 6s ” 
1974 a 16 33 a 
1973 “a at «& 6s 
1976 “0 6 37 m 
1977 “ Ty 2 a? 
tom 4 2 " n 


$969-28 eggregaie = 08 17 3a $30 
(949-78 average “0 


has again declined more recently as a majc 
has. been: put intn the new eli descane radio- 
linked Interferometer (sce table 4 ’ eve 
afte; slowing for these temporary fluctuations; the 
average level of scientific production of Cam- 
bridge has still been-considerably higher than that 
of the Jodrell Nank group over the ten years up to 
1978, 

As we have already seen; the rumbers of publ. 
catlons ia bit a partial Indicator of scientific pro- 
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gress While ihe erence between the f pul 


cate * difference i in scientific Progress. it may al ale 


he accounted for hy other factors. For example. it 
tay be related to the different types of ratio 
telesctipe and research interest a1 the two centres. 


Alicrnatively. there may he substantially dilfercni 


Lad 

result that Jodreil 
make a more substa: impact in-terms of their 
contribution to scientific knowledge. The latter 


possibility is examined in the next section on 


able’ | el ee 


Nerbers of publications relative to stalling wad fending teveie® 
be torn 


: Cam Inde = NFRA MPT 
beidge = Bank * 


* ‘The likely errors in these figures are prohably somewhere 
between 5 and {0% (see mote a to table} anf ante 2 to 

- table S.- 

* These ave the figuees fow.a single yer only (1978). and hn.ts 
Enectivahle thei they might he wntypicel of the decade 
1969-78 Stowever. for the tesstms mentioned in mate 3B. it ts 
waliketly that figures few the four centres would show a very 
different pattern in prevines yeors, at beast once the ten 

- Continental centres begen operating at fell strength, 

* The. figuies_in. parentheses are those obiained for Jodiet 


apent on teaching tece table 2). 
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t the Tonver can he considered here by 
1g al the naire of the Dutch and German 
Rrowps. = 

During the. ten years, ur to 1978. the Sear ner 
of papers by both the futch and German groups 
has increased rapidly, finally reaching a level about 
twice that of the Cambridge group and. more than 
3 thac of Jodrell Rank. Tits high volume 
of production for the two contincnial groups can 
he partly explained in terms of their higher fund- 
ing and staffing levels, If one allows for this (sce 
fable ne it appears” that the 2 of the 


and Jox rell Bank groups ( (even after allowing for 


ment-building = sce-nate-¢ 10 table 7). Whether 
the smafler number of publications | for the latter is 


compensated for by each publ 
On average, a greater contribution fo scientific 


knowledge is considered in the neat section. 


the Science Citation Index: These are detailed in 
table 8. The number of citations to Cambridge 
work has increased by over 150% during the period 
gud with a slight pause to this. growth 
evident i 


for citations to- ok Published | over the prin 
four years, suggests that this temporary {all can 
Probably be attributed to the diversion of effort at 
the beginning of the 1970s away from research and 
into the commissioning of the new $-km- inter- 


ferometer. *? Both the Dutch and German groups 


the numbers of 
tions_to their work, reaching. in 1978, a- total 


Similar in magnitude to that for eee In the 


*Y The effeers_of the enmceniration on Jastroment-bullding 
shee up even more cleatly in table 11, which indicates that 
the. citation sate. per Cambridge. publication deopped 
markedly tn the early (970s. During this pertad many of the 
ata{f were bavolved in building, testing. and onmeniasioning 
their mew tclescrpe, sad, while the publication sete of the 
group 03 ¢ whole Fell only slightly (largely kept wp by some 
highty productive Pu stedentt), the citation cate far each 
porer declined dramaticaliy. 
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Tae Tenet 
Citations wy all previrwe work area Rates of acti citation ESC» pend “in nese” citation ICs in 
io ton 
ort = a = eo 
Cam deetee =O NTRA MPT 
beige Mant 
Namber-of citstions 
_ analyred in - = 
320 __saenple” 207 Pr] 200 200 
$30 No, of SCs os 4 » 4 
tox0 No, of (Ce _ 32_ 2 Me Le 
1978 Citatinm to _ si mee ete pare Percentage of SCs - at 2 ord 8 
1969. TR emt only 1120 Man {yan ~~ Inyo Percentage of ICs (se ut rt Thy 
————— Tote) & of SCa and ae _ a 


* tt be difficult to strsch any ectimaic te the Nikely errors in 
ihese figeter (1 they were of ihe coder of Ya thes enwld 
correspond to percentage errors of between 3 and 7%. 


case: of Jodrell Rank: however, the. number. af 
Citations “to previaus work: has scarcely changed 
during the §2 years studied. Similarly, table 9 


ak iat Geen Goes ta oa ae 


than that by Jodrefl Bank. 

- In -the ‘earlier theoretical discussion;_ it: was 
argued that citation rates should be. regarded ax a 
partial indicator of scientific progress. This being 
So, we mist Now. cxamine the extent to which the 


tower citation rate-lor capigl Bank reflects a 


such “other factors” inciode? po for example. 
the tow citation rate be altributed to weaker links 
with the international community of radio astro- 
-Womers, less of whom therefore dite Jodrell Bank 
‘work? Scientists in both Holland and Germany 


commented, in interviews, that both - Cam. 


tohies 
Citations to work pobtished in the previews low years. * 


Combedge Fadel «NEMA MPM 


piesa Bonk 
(970 ciistlons = 340 200 
1971 cliatlons : 20 = 
(972 clatlons fee = yaa 
1994 chations 990 20060 4 
3978 chations «= $50 to ee 


* See nates tw table &. 


ANC: qhhety Wet 6E kot 
ereen) (Os) C298) 8 Ley 


© Thie comple of 197% citstinne was chee am 2 random bak. 


bridge and the Jodrell: Mank groups tenit ta he 
som-what “closed™ in their attitides towards owt- 


siders, but ha claimed that this tendency is more 


| seem ¢ bute the 
tatter's low citation rate ta this cause. Can the 


Hence, it would scem difficult to att 


differences in numbers of citations for each group 


he altrituted to differing rates of sell-citation and 

“in-house” citation ‘ * amiongat - the centres? A 
sample of about 200 1978 citations for each centre 
nined, and table 10 shows that there is 


in_the ane with which they cite their own 
group’s work. 

_ Yet another possibility is that the total naraber 
Of citations for the work of a group is very. largely 
determined: by its publication rate. We can ex- 
amine whether this is a satisfactory explanation by 
fooking at the average number of citations-to each 


paper. Table Hi shows the figures for. the Citations 


divided-by the number of publications produced. in 
that time, Le, the average number of citations per 
publication (CPP). From table 31, it can be seen 
that the. Cambridge group. while gre 0 lar 


igher number of citations per | abticiion, 


This suggests that the number of citations is not 


just a Function of numbers of publications, 


ae 1c thal aide alee @ cle cae 
her grep, 
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Tatte if 


Citations per poticniton (C¥t) for wot pobished in the ; 


Previous fever years 7 
Com fedrefl «NTRA Pt 


groups have made father more impact on the 
advance of knowledge than Jodrefl Dank-in the 


perio 

particularly well in relation to its size. The fig figures 

for citations per publication aRain suggest that, of 

the: four: centres. Cambridge papers have @uerally 
impact (with the exception of those 

at.the staet of the 1970s - sce note 43); in this 

respect, Jodrell: Bank seems to have done slightly 


bridge = Bank 
33 2$ ae 
22 
CTP in 1972 : a3 
CE in 1994 - 21 23 24 19 fxs 
CTR inten 32 28 24 ~ ~20 


better than the Dutch and German groups, al- 


* The fikely erows in. these Figures probably range between 3 
and 7% (sce note a io table B) 


There would, therefore, seem to-be some justifi- 
cation for arguing that, for the radio astronomy 


ation rates-do provide a 

ful_ pa jor of scientific progress, In 
the case of total eset rates, the figures certainly 
suggest that the Cambridge, Dutch; and German 


abt 12 soc | 6 
Mighty cited pepens peblichel hetween-1909 ond 197% * —— 


Ge ede NERA MPD NFRA MPI 


t et te 
] 9 () 
° st oe 
yo ah 


i foctudes one review paper. vad one paper describing the new 
tynthesis telescnpe. 


though-the soral impact of all its papers is ap- 
parently still much smalies because of its lower 
publication rate, However... until s we fake account 


ring | the. peer-evalt ation results, 

however, there is one. further argument that could 

he tevefled agsinst.the preceding citation analysis, 
i nised. 


( 1 

n-of human knowledge) w 
Comiribuies most to scientific progress, but a snail 
number of key papers, each of which has a_very 
great impact on the advance of knowledge. If this 
is the case. then the figures for total numbers of 
Citations, and {or citations. per publication, may 


not reveal which centres have managed to produce 
such key papers and which have not. However, we 


most highly cited papers-and, 
secn earlier) there are problems in using citations 
for comparing individual-papers, this should give 
some indication as (o the distribution of key Sapers 
between the four centres. “? Figures fot apPproxi- 


Ep operon erring rr rer prckis orm 


ot feast » parti indicator of the number of “discoveries” 


made by 0 erscarch group, and there i iadeed some-evi- 
dence to suppor! this eeppaiition free the peer-evalustion 


* Rigoroes efforts have been made to ensere that these figures 
are as sccursie as possible. 11 is enlikely. that-any of the 
valves_ia the fiexi_two rows is wrong by more then ! of_2 
wns, Ertors of this magnitude would have no effect on the 
concietions drawn in the tert concerning the distiThution of 


if 
iy 
Bia 
i 


fie. abowt 1.22) have heen cited 15 or more thes in any one 
ear. 


taterviews deacribed later. In these, astonomens were asked 


to Kdentify the principal “discoveries” ot significant conte. 


Swilows to scientific knowledge made by each of the research 
Groups aver the ‘ten years up 10-1978. There was a high 
Sepree of consenres among the interviewees on at least the 
major enniributions, und these generality coincided with the 
moat highly cited pepers. Moreover, at one of the two 
centres with very few highly chied papers, many of the 
sponsible for ao major discovery dering the ten-year period. 
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mately the top t& most cived papers (cited 15 or 


more times ig any ane year) ate shown in 


Ut can be seen that between 1969 and 1978 


Cambridge group produced 
and the Dutch Brom hall th 
a 


have diminished rapidly: the seoned row of table 
42 takes_secewnt of the variation between, these 
short-lived Papers and those that have been highly 

ied aver many years hy showing the aumber of 
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order according tn the magnitnde of their coniri- 
butions | ter . scientific knowledge made over the 
I come ape es 


each cenire, and the consistency between centres, 


In particular, we Inoked for systematic d 


one's own centre. a 
Fic seengerelback (ema scale hetween | aed 
9" because there are nine centres). obiained at 
each centre are shown in table 13. A 
comments can he -made- shont the scsolts, Firs st 


there papers managed to achieve 15 or more 


> : ferences between the tour groups 
are. if anything.- further secentunted. (Lest it be 
imagined that this distribution of highly cited 


Papers ix in seme way the retult of an 
athiteary value of 15. citations a year ax. the divid-. 


remulis obtained hath within cach centre, and he- 
tren the centres. Indeed:-if one averages the 
results forall (our Centres. the great majority a 
out of 3%) ) agree © na 


ing line between highly cited and less highly cited 
papers, table 12. riso incindes the numbers of 
papers: cited 20 times ar more, and 12 times or 
more, in exch case = & similar distribution 


13. Peer evadeation 

- In it series of pains 70 interviews with 
vittwatly the entire scientific staffs of the. two 
Firitish groups, and-with shout. one-third of the 
1 German groups (including all the most 
senior astronomers), scientists were asked to Sden- 
tify the main contributions_to scientific knowledge 
made over the period. 1969-1978 by their own 


cenires. “* This group included ail the radio ob- 


servatories considered by the astronomers in our 


survey as being the foremost centers in this field. 


They were then requested to rank these centres in 


eowrsate ( the scientific progress made by their 
awn group. but while thi is significant in two of 
the four centres. *-it isnot as marked a might 
have been expected. and docs.ant appear to be 


of certain particatarioGc factors on these neler 
For example, Cambridge ratio sxtrancmers tendad 
to rank interferometer centres slightly more highly. 


"17 ka eight be eapecied. dhere aaa 5 Geriske vebecinnct io. de 


this, grecrally en the grounds that, becawse each cenise is 
lnvedved in slightly different werk, i be misheading tn rank, 
them on a single scale. Nevertheless, murat reapwrndents. 
(88E) id. ogvee Wo tate part bn this cambing crercice: seme 
teh atle tn sank each centre bn ander from ¢ in 9, while 
erthers preferred to growp the cemtres ime categories of “First, 
Clam”, “necrmd clan”, ~ thied clam”. etc. * 

* Uf ron centres were ranked first-cqual, they were given the 
sverege ranking of (.5. Similarly, 1 three were ranted equal 
Gast. they were given the tanking of 2 tthe average of t. 2 

_ and ST. and on on, Z 

&* Fewr of the remelning five differed hy only 1.0 wnits, ond 


“* Meshtes the feet centres already discwseed, these tncteded 


the remaining ane, where the diecrepency wm 2.2. "a8 8 


the Americon sadin televenpes of Arectho, Cohech, ond the 
Notional Ree Asironneny Citmervatory_ (NKAO), _ the 
Freach centre a Nangay, and the Anstration big dish of 
Parkes, We hove, a3 yet, ant inoked of peblicution and 


Chatlon Indicators for five ofthe nine centres, but bape in 
dn 20.in the future t0 see how the renults.correfete with the 
teselis of our peer evaluation, This eneld then cnasilivic 8 
atedy of a large part of the exrid redin attronnmy crmnen- 
lty, and the teswhts wovld be of relevance acd enty Ie ectence 


froticy, hut alse te the anciningy af acience. 


ws 


satf-conking — le. the ranking by one centre of its own 
% tn sdditen: she neakow MP otal tend 50 slighty ves -nte 


aclemiific peegress of the grown. This Aiflerence between the 
evaluations of_senien and juniog stall at the Germon cratte 
may he releted 0 § certain dissathiaction amongst the latter 


atoet the organization and evicatation of the group's re- 
srerch. 
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Tahki i 2 i 

Rankings of wine taten eterncmy centres * chained by prey evstuation (FT) ~ Se a de os 
Tha | frat hat A 
Jodrefl_ flank NFRA_- MPT __ 
(#21? (Ch A) eee CE 
73 79 67 7.2404) 
to 67 $3 $.7(208) 
3 9 F 3 2.2 (208) 
38 64 6t 3.4 (3 10) 
43 43 5.3 $.2(40.7) 3.0 (20.8) 
(x) ts a9 68 (404) €2(201) 
3 16 2 25 (20.6) 2.8 (20.3) 
43 28 26 32¢207) 32(20.2) 
36 37 Sé& 47409) 47040. 


5° me aanaple alee 


© embed bn terver of the magnitude of their conteitutinas ta Acientific knawlcdpe hciwech 1960 sind 1978. The iguccs fa parcmibeies 
im the final two columes Indicate the ront-mean-equare variations between the rankings given by the different centres. 


and some of the big:-dich centres less highly. than 
Average. stigecsting that the type of equipment and 
type of research at each centre may al affect the peer 


‘tended to rank-that centre-1 ty 
age: For example, some. Of the American astro- 
“pomers wer! ing in the Dutch and German groups 


ranked the American. fadio. - aatTENOMyY centres 


However, our results suggest that, while this may 
be tre for: some individuals, the effects of these 
particulsristic factors are barely Sigaificast faa 
ane fairly hal abil of baa reviewers. °* 


* The ‘American _sstronomess working “in Holland - and 


average (91.6.4 (4 1 8)_compered with the average of 5.7 
(2.0.8), bet again the difference le probebly ant significant. 
Recawse of the relatively small sire nf this systematic devie- 


Hon, 1 seems dowbtlel whether the overall pear-evalention 
results would be substantially ahered by incheding radio 
ie ck ce 


thas of French, American, and Ausiraian redio astro 
Romer, end this arey be attempted bn the fetere. 


The high ‘degree of consistency i in the peer: 


at t feast: when carried. out with a large 
representative sample of ‘scientists from carefully 
matched groups, can yield a reasonably good indi- 
cation of the scientific a by resean 


placed in the peer-evaluation seid, hat t do the 


results actually show? The average rankings of the 
nine groups are shown in fig. 3, 9? these averages 
having been obtained after excluding self-rankings 
on the ground that these may sometimes produce a 
small pe nevertheless: Significant. effect on the 


bridge 
National Radio Astronomy- Observatory (NRAO) 


in-the United States, are clearly perceived as hav- 
ing heen world-leaders in radio astronomy over 
the period 1969-1978. They are followed. by « 
group of four other observatories. including Jodrell 
Bank and Bonn, with the former apparently in the 


lower hall of this group. The two rema ning grou; 


ss {mest be emphasised thet this ls neta graph: K lal increly a 
pictorial representation of the wemerical results in the right. 
bend. entuma of table 13. The nine centres have heen 


arranged In_the-erder in which they were conked-solely to 
bring out_the differences perceived by radio sstronomers 
hetween the top three nbeervetortes, the second group of 


(oar. and the remaining two cenires, 
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: astronomers mers tn. agen anys 


rs FA Barve and 3 Sovewe  Aneveing haw reicmeh 


over r this period: ** OF the four 
¢ principally concerned. il 


can Be secn yr the Segal group is re 


ientifie progress 


percep crs). 
his | important fo note that fig. 3 presents a 
static: it does net show changes in the 


relative positions of the centres over time. Resides 
ranking the observatories in: terms of their perlor- 
mance over the ten-year period, we did atk radio 


hanges in position 


highly successful over the preceding nit Haw. 
subsequent years, there appears to have 

it in the: relative 
1 of the two newcomers, 


NFRA and -MPI (in line with the changes over 
lime noted in the indicators discussed earlier). This 
improvement came at the expense af Parkes, Jodre™ 


Rank, and ale ie sulfered fee. the in. 


ee eee 


challenged first. hy * 6 'Deich group when their 


new interferom=:cr began operating. and-is now 
about_to face a severe challenge from NRAO's 
new Very Large Array. And perhaps_in the near 
future, the MTRLI (ace tahle 4), which was under 


_ Consirnction in the Beret up to ioe may restore 
ne 


of spate sly 19004 by Todrelt" Doar has in: 
creased ‘significantly since this work was carried 
out (10 20 in 1979), and the extent to which this 


” Sep een inal aie Gee ee 
omy groupe were mentioned. as-having achieved significant 
sctentific progress. These lnctuded the genups of Flolngna tn 
Linly, Rentieton in Caneds; ond Bell Telephone Leboratorles 
tn the Cinited States, Some of these groupe, Hf they had been 
incladed tn our fist of observatories, might have ranked as 


highly 28 the middle group of Parkes, Caltech,.ond Jodrell 
Benk. perheps pushing some of these centres down the 
santings, Nome. however, were scen aa being In the sarne 


fank as the top three observatories. 


1969-78 (exchuting sefl-rankings). (The erroe-bars indica ee 
foot -mern-sqnare variations between the rankings given by the 
Gilleren centers.) 


kivowledge relative ta thai made by other radio 
observatories would be an interesting subject for 
future study. 


m Dhcmeton and para 


comprehensive, | i 
industrial soctetics, Sony peas amounts of stave 


expenditure ‘are now concentrated on a smal! num 
ber of “Gig Science” centres. Perhaps the key 
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makin in the > past + hae een ie lack of an 
adcquaie methodology for evaluating “success” jn 
basic research. In the theoretical discussion in the 
earlier part_of the paper, a.cer analysing the lir 
tions of such measures ax p.sblication counts, cita- 

“diacovenes™, a 


factors” on. the Partial indicalors have been kepi 
relatively sniall, and. hence that some reliability 
can he placed in assextments hased on these four 
indicaiors. So what do these four indicators sug. 
gest for-each of the radio astronomy centres? The 


results of this study suggest the following conclu- 


af partial indicators for comparing: the scientific 
contributions made by rescarch groups working in 


evaluate the performance of the two principal ra: 
dio observatories in Britain - with NERA and 
Donn brought in as ste major contro! groups 
There sre just (wo ques'ions remaining to he ad- 
dressed in this final section. First, 4o what-extent 


are we justified in using the methodology de- 
scribed bere to compare: and to diflerentiate the 
scientific performance of basic-rexcarch groaps7 
And: sccondly, what_policy implications c.n te 
drawn from the results obtained in this way? = _ 
The most important thing to note is that there 


isa high degree of consistency between the results 


Cambridge 3 ind 1d Dutch groups have been relatively 
large; and, if we allow for the differences in scale 
Of research ‘activity between the ‘centres, the Iwo 
partial_indicators based on publications per re- 
searcher and citathons per_paper °* suggest that the 
research productivity of the Cambridge group has 
been the highest of the four-centres. This conver 
gence in the resulis. suggests”? the selection of the 


Control groups for comparison with the two British 
Centres has been such that the effects of the “other 


1 Tis ioe the lndlesies = sel semen ct. publications 


torn of scientific progress. For exemple, both veggcst thet 
the- German group hes achieved as much as the 

and Duich groupe. ‘The resion fs thet aumbers of pubtice- 
Hous tefloct the level of acientific activity eed prodectinn of 
a research contre rather than the scientific progress ts 


making, while o Serge. total. member of -citetions can be 


Bencraied hy pobitehing & large sumber of paren. even Hf 
cach mokes Uitle hepect on the advance of scteniiixe know! 


5; odee. 
** Convergence does not, of courss, “prove” that the indicators 


; sions for cach of the cent:cs. 


a) The. Cambridge radio asinzsomers Bre per- 
ceived by their peers = naving achicved. as 


100-1 Say ee a es oe 
1969- 


groups, this disadvantage i is offset partly by 
the Cambridge scientists schicving a high pub- 
lication rate per reseatcher (especially if ane 
aflows forthe time taken up with under- 
Graduate ‘teaching), and parily by producing 
Papers which, on average, have more- impact 


on the advance of scientific knowledge. They 


have also produced 8 -comparatively large 
number of highly cited papers, * s result which 


is consistent with the judgement of their peers 
_ that they have made several major discoveries. 


imuch, or. almost as much, 88 their Cambridge 
Colleagues é in terms” as contributions to scien- 


in. turn suggests that cach Dutch 5 peutioeis 


does sae constitute-such a substantial gontri- 


in significantly fewer, highly cited papers and, 
according to their peers, fewer discoveries than 
the Cambridge work. ; — 
he large number of radio astronomess_ at 


. despite a somewhat lower 


publication rale per person, the group ass 
whole produces a large number of scientific 


tal 510% oi Comtlligs papam achdeved tl av mora cliations to's 
yest. .compered with 0.9% for the Deitch group, 05% far 
Jodedé Goch, and 0.2% for Base: 
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jrulged by. peer: evaluation, This ms S he inter- 
preted as x implying that cach_German publica- 
. has a rather smaller iaapact 


Jin per 
In addition, the Ronn group has 


not produced very many highly cited papers. 


nor many “uliscoveries” in in the opinion of their 
peers. : a 

(4) In the case at Sealeell Dank: athe. prihlicaGon 

tate Tor the growp as a whole is relatively law 


ate teaching and forthe -diver- 


sample_of radin -astronnmers ‘us 
evaluation process was not 
wards interferometers | (we actually ee 
tather more hig-dish than. interferometer. astro: 
nomers), the peer-cvaliation results do soggest 


employed. a vin view-nf the fact that the 


that the effect és more real than artificial, and that * 
the perind between LOR and 1978 should ennse- 


sion of effort intn the new interferometer pro- 


ject. This somewhat lower level of production . 


is partly offset hy prodacing papers of higher 
impact than. those of the Dutch and Germans. 
Overall, however, the Soma th scien- 
tifte 
tremomers -are judged by their | peers to be 


significantly greater than | those of their Jodrell 
Bank colleagnes and thos 


given that in. terms. Of effective number of 
researchers (sce tahle 2) ihe German group 
over. twice that of the Jodrell Bank tea 


quently he regarded as the era of the interlerome- 
ter in sadio astronomy. | 

Another factor that can influence the scientifie 
performance of-a research centre is the fraction of 
tine its. research stall devote. to instrument-build- 
ing, a Fraction that may vary cyclically as sew 
teleseapes are brought into operation. Tn a certain 
extent, the-relative performance -nf. Jodrell Bank 
nver the period 1969-78 may be linked to their 
concentration on ii t-building during mech 
of this period. - 

However, we shall leave a discussion of the 
teasams for the “differences in scientific perfor: 


seed ag’ lfference evident in the numbers 
Of highly cited papers — neither group has 
fietlormed ‘very well in this respect relative to 
NFRA and, in particular, to Cambridge. 


aper to make specifi 
individual centres. That i is 


hic for the funding and organization. of basic 
research. The. decisions they reach will he depen- 
dent u upon other factors such as the availabic fond- 


sures tn improve the performance of the less suc- 


centres. For example, one factnr which has 


apparently. influenced the scientific performance 
nf radio observatories Is the type of instrumenta- 


peek 
fe ont 
GO! 


significant differences in | 

of radio astronomy centres he Identified? We would 
contend thatthe evidence presented in the paper. 
Sufficient to justily.a positive answer tn both these 
questions. However, it should perhaps he cmphe- 
sised that. we have been epoca? with the assess- 


clearly interested in the ian Performance of 


research groups. Nevertheless, past_performance, 

although by no means the only factor, is nne of the 

best_indicators of. Sulure Performance, particularty 
Ti ro e rtua' 


Sct Up & new centre or a major research pre 
gramme overnight. We therefore see it as seclal 


that those who have to judge the respective claims 


arch groups, do so 
the hasis nf full information on the recent pestos- 
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mance of those groups. Although further work i is 
obviously needed. we helieve that this paper pees 
some way in demonstrating how that information 


canbe obtained. Ww extahlishes a methodology ihat 


in their-owan rage and which, if other a 
are equal or indeterminate, should he given a 
relatively high priority in their claims for research 
hinds. 
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5 What Direction for Basic Scientific 
Research? 


John Irvine. and Ben R. Marcin® _ 


Science Policy Research Unit, Univarsity of Sussex 


currently seems far 
from- secure. -- In most OECD.-countries, rapid growth rates-in | 


INTRODUCTION 
eat 1 


The. future for basic scientific research 


from this trend. Initially this gave rise to demands for 


magnitude of the wide range-of outputs from different -scien~ 
tific activities, such policies inevitably receive criticism 
rent; long-term 


New mechanisms for decision-making are. needed_if-the- direction 


of basic scientific research is to be put ona. satisfactory 


* No order of seniority implied (rotating first authorship). 


The authors are Fellows of the Science-Policy Research Unit, 
University of Sussex, where they work-on a range of issues-- - 
concerning government policies for scientific research. The 
work. reported-here was funded by-the Social..Science Research 
Councs1.: Thanks are due to various colleagues, in particular 
Jay Gershuny, Dot Griffiths, John Krige, Ian Miles, Geoff 
Oldham, Keith Pavitt, Tim Shallice, and Tom Whiston for their 


comments on earlier drafts of this paper. 
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basis in the future. . But what are the various types of deci- 


sions currently facing those concerned with science policy- 
taking? What methods. and criteria can be used in making 


them?_ What problems are associated with these methods? And 
to-what -extent can. such problems be overcome, so as to yield 


more acceptable and hopefully more effective policies for- - 
Scientific research in-the future? This chapter sets out to 


suggest ways of answering such questions. 


RESOURCE ALLOCATION IN BASIC SCIENTIFIC RESEARCH 


In allocating resources to basic research, decisions-are ----- 


required at a number of levels: Starting with a total state 


réd budget, these involve asking: 


1. How much overall. should the. state spend - on- basic science 


compar ed with other areas of r&éd such as the more applied 
-- geclences-and-engineering? - - - = --.--_--- --- 
2; How. should this overall budget be diseribated Between 
different (basic) sciences such as Physics, ‘chemistry, 


3. What should be the distribution between different - speeiai- 


: anit: between optical and radio 
sics, solid-state physics, etc., 


rae Within each: area of science, how should funds be allocated 


between. different types of. activity Be ego 


5; a Ba ioe be allocated ta ace: Se ck centre; 


group, -or- -individual, -working withiti 4 given specialty, 


both to optimize current research activities and- ensure the 
maintenance of an effective research effort in the future? 


PROBLEMS IN RESOURCE ALLOCATION 


The- -eoming- ‘crisis in. basic research was to some extent antici- 


pated in the early 1960s by the eminent physicist, Alvin 
Weinberg. -Weinberg pointed Out the seeming. absence of any 


underlying rationale to resource~allocation patterns in basic 


Science. - In-what became-a-seminal paper. in the science 


policy. literature, he distinguished between two sets of. : 
criteria --internal and-external‘-- that-he advocated should 


be_more widely applied to determine the relative levels of 
Support for different scientific activities.- -Internal -- 


criteria relate to the impact the research is likely to — 


on the advance of scientific knowledge within the field con- 


68 


pemai 
eb a 
r worn 


o.. 
ERIC 


147 


the decision-making process3, and,-as-we-shall-see below; have 
tended to-reflect the values and commitments of certain estab- 
lished intepest. groups. In contrast, the distribution of 

resources within fields and specialties appears, in the ——_- 
Western world_at. least, to-have-been made largely-on the basis 


of internal criteria.. Such decisions have tended to be the 


knowledge, and hence to decide between the competing claims of 

different researchers for funds. In the words of Polanyi: 
"So long as each allocation follows the guidance of scien- 
tific opinion, by: giving preference to the-most promising 
scientists-and subjects, the distribution of grants will 


automatically yield the maximum advantage for the advance- 


The fact.that such a justification entirely begs the question 
of what is meant by ‘promising’, and how it is judged, has all 


The principal mechanism for the organization of basic scien- 
tific research-has thus-been based largely on internal regu- 


lation. - Politicians and civil servants have normally been 
content to leave such-matters to the scientific community, 


provided, of course; that the resultant activities are in 
general accord with the external criteria implicit in - 
decision-making at government level: However, the pressures 
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problems with. internat mechanisms for the cenntatton of scien- 
tific activity. These problems have been accentuated in -many 


countries by government attempts to introduce stricter and 
more- explicit external criteria at- -increasingly lower levels 
in the decision-making process. . It is therefore time that 
stock is taken of the- -strengths and weaknesses of internal and 
external regulation of science as currently constituted. 

Only on this basis can we consider how these mechanisms for 


résource allocation wight be iaproved. 


deguiatton of actencific ceveacch by tnearnas criteria fas 


traditionally been based almost- entirely on peer-evaluation. 
Peer-evaluation has been described as: 


‘a ikind. of science. advice involving select members of the 


up 
biases and, probably, on sheer caprice as well". 5 


Despite criticism from m chose scientists unabie to pass the 


Weinberg’ 8 criteria for scientific - choice (see note- 26- -Ta 


part, this is because flexibility to accommodate demands for 
resources from-most. established scientific-groups-and-disci- 


plines was built into the system by the rapid expansion of 
RBational science budgets between 1945..and the early 1970s. 
It is clear chat there is far less. strain on a system admini- 


stering- steadily - increasing budgets than one oriented towards 


restructuring and reducing expenditure, where ‘gatekeepers’ 
are put under severe pressure to abandon their-professed - 
disinterestedness aiid openly-defend (or attack) individual, 


institutional and specialty interests. 
Indeed, it could be argued that the whole infrastructe for 
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allocating research resources -on the basis of peer-review is 
now under: such pressure that it is-in-danger of breaking - _ 
down... There-are three main reasons for this. First, the 


long-period of post-war growth in basic research activities 
has. led to-the-entrenchment of-particular interests in 


areas.of natural science such as radio astronomy and high- 
energy physics.’ Their prominence inthe war effort, and 
the social and institutional. links they had developed with 
politicians and civil servants, made it relatively easy to 
obtain funds to build-large research facilities such as 
radio telescopes: and accelerators. These research areas 


were therefore. able-to grow rapidly; and to become what is_ 


now known as ‘Big Science’. Today, for-example, one major 


alone is currently planned to cost approximately 500 million 
dollars. 
Most. importantly, as the level of funds for Big Scienee_grew 


(to over 50 per cent of the basic research budget in some 
countries during the 1960s), so did the répresentation-of 
their respective research-communities-on- the bodies respon~ 
sible for-distributisg the science budget... In the case of 
Britain, the situation-is now such that, within-the Science 


and Engineering Research Council, the organization respon- 


sible for funding-research in- the natural sciences, one sub- 


section of physics, nuclear Physics, has its own-Boara,; as- 


does astronotiy_and space research; while all other zreas. of 


Science -- physics, chemistry, biology, mathematics and com- 
puting - have only one Board between them. The result is 
the situation depicted in Table 1: Big Science consumes: 
almost double the resources of all the remaining natural 
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tion of total sctentific activity; at yer in terms of 
numbers-of doctoral studentships and postdoctoral research 
fellowships: 


with resource distribution 80 iuavily dependent < on. decisions 
made by scientists themselves; there has inevitably-been-a 


tendency for certain early established sets of priorities 


and -research: interests to become- 'frozen into‘ -the-decision- 
taking structure... To those on. the periphery of this strucy 


than a means of ensuring ciate those funded generously in the 


past: continue to- be supported generously in the future.% -- 


Whether.or not this judgement is accurate, it does point to 


a key_ problem associated with-the scientific community 
itself having main responsibility for decisions on che 


distribution of the basic ‘science budget. This is that the 


eon 


ticular soeetaley or science depends to a great extent on 


will. continue to: be stanitle supported Soe ‘they are well 


represented in-the-committees determining funding 


priorities. - In:addition, such-areas_have-often been able 
to. institutionalize large fractions - -o£-their-- research. 


new and potentially important field may sometimes find 
difficulty (unless established scientists lend their 
support) in eliciting funds because there is no rélevant 
committee (the committees-tend to. reflect: prevailing pat- 
terns of scientific activity) or because they aré poorly 


represented. 


only very slowiy,-if at all.- -Conversely,- scientists - in a 


The second. major problem concerns the effects in certain - 


research activity within fewer and fewer institutions.- As 


these areas have developed,-so they have come. to. demand - 


increasingly sophisticated and expensive research faciiities 
(astronomers now have access to space-borne telescopes, for 
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Table 1: Some Features of British Science and Engineering 


Research Council Funding of Basic and Applied 


Science, 1981/2 


Area of Expenditure! Budget 


"BIG SCIENCE: 


Nuclear Physics 44.0 

Astronomy, Space and ak 
_ Radio... 3731 
Total Big Science - 81.1 
ALL_OTHER BASIC AND APPLIED 3 

SCIENTIFIC ACTIVITY 4436 

(coming under the 

Science Board) 
of which: 
SERC establishments” . 16.7 


7 
International contributions 7.3 
Other domestic facilities 1.0 
Research grants to: 

Biological sciences 6 
Chemistry _ 7 
Mathematics -- 0 
Neutron_Beam. Physics 0. 
All other - physics 4 
Other sciences — : 0 
Central support schemes; _— 
Projects and administration 16.8 


AND APPLIED SCIENCE! 142.5 


No . ae 
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Studentships and 
Fellowships ciirrent 
in 1981/2 

Doctoral Post- - 
Research doctoral 
Student~ Fellow- 
ships ships 


223 26 
399 33 
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Source: Adapted from data in 
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2. Much-of this is for the support of two SERC establish- - 


ments recently re-oriented from high-energy physics, but 
still -working. in. areds- requiring the provision of large, 


expensive reszarch facilities (e.g. synchrotron radiation 
and-neutren-physics). whose non-inclusion as *Big Science’ 


is therefore a matter of definition. -- .- --- 
3. This excludes-an allowance of £3.3 million for 
engineering activity. = © 2222 cu 2) fittt. Le 
4. These are funded by the Joint SERC/SSRC Committee, the 


Polytechnics Committee, NERC, etc. : ; 
sport of the Science and 


Engineering Research Council for tne year 1981-82 


(Swindon: SERC). 

(astronomers -have now.access to space-borne telescopes, for 
example)."--Since it is no longer possible to provide 
separate research: facilities for all-groups working in a 


field, there has been a trend towards siting major instru- 


aséociated user-groups. Moreover, the problem is not con- 
fined to physical science: in-medical science, for example, 


there are similar disputes over the relative emphasis that 


should be given to research institutes and units compared 
with university departments. 


It is; howevér, with the political effects of such concen- 
tration.that we are most concerned here. -- In-a specialty 
like high-energy -physics,-where formerly there were a 
number of university groups, each operating their own 
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1. 
2: 


5. 
6: 


7. 
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Resea. ch Council Expenditure on Basic and-Applied 
Scientific Research on Large Central Facilities, 
1981/. 


£M 
CERN (High energy physics) 22.5 
Rutherford Appleton-Laboratory. 
Qiixed facilities, including-central 
laser-facility; spallation neutron 
source, high-éiergy physics support, cawe 
and space) 40:0 

, Institut Laue-Langevin 

(Neutron beam facilities) 7.2 
Daresbury Laboratory 
(Synchrotron radiation. source, - 22 
and -truc lear structure facility) 16.2 
European Space Agency 8.3 
Royal Greenwich-Observatory a 
(Optical and infra-red astronomy) 8.5 
Royal -Observatory Edinburgh _ 
(Optical and infra-red astronomy) 3.4 
Total SERC expenditure on these 7 Le: 
facilities _ 106.1 
Total SERC expenditure on all science 
(excluding engineering but including Bes 
postgraduate awards) 164.2 


Note: 1. The figure for the Rutherford Appleton 


Laboratory-excludes about £8.7 million which 


was spent on engineering research. 


Source: Report of the Science and Engineering 


Research Council for the Year 1981/2 
(Swindon: SERC); 
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accelerator, there are now-at wost-two or three main - 
research facilities available to the scientific community of 


any one_country. - This poses a threat to the traditional 
teans-of determining-research priorities; the successful 


operation. of peer-evaluation depends upon the existence - -of a 


relatively independent expert quiseneac es. --In other words, 
there must exist-sufficient scientists-familiar with the 


research area where support is: being considered, but whose 
own -material-circumstances-will be unaffected by- the vutcome 
of the decision~making_ process. - When. the. number - of dis- 


tinct research. groups working - in-a-specialty-is relatively 


large, this condition is at least approximately mets: 
cating + resources to:a major centre ig potas ceased ved” 
nearly-all._peers-will be-a user either-of that centre. (and 
80. wilt. _benefit from a postue ‘Secision) », or of another 


might privately chink of the merits of chowe pe eee eee 
taking it in turns-to seek funds. They thereby present a 
facade of unity and consensus on priorities: within che — 


peting for funds vith other specialties. Instead of the 
"free warket' of scientific ideas - all competing with an 
eqiai chance of funding - the. notion on. which sciertists 
have traditionally based their-view of peer-evaluation as a 


neutral disinterested process _ there has een a tendency 


dominant. influence: on funding-wecbanisus.- - : The: open. and 
*objective' -seientifiz - community- (which, - Like-the -'free-- - 
market'- of: classical economics, is probably more-a.histori- 
cal-myth than an accurate- representation -of -reality)- has 
given way to a decidedly hierarchical. system where the 


distribution. of resources increasingly pecms to obey the 
maxim, ‘to them that hath ahatt be given’. 


A third and 1 direcrly retuced problea concerns the general 
ineffectiveness of peer-evaluation as-a-mechanism for 
restructuring scientific activity. _ This particular. problem 


ha~ only really become apparent since the science budgets of 
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Western countries ceased to grow. _-Science-is inherently 


dynamic..in nature, with new areas. of ‘activity continually 
emerging and other older areas reaching a point where fur~_ 
ther research is subject to diminishing returns.---The long— 


term vitality of -the basic research system, and indeed its 
Capacity to.provide the knowledge and skills to-help solve 


the pressing economic and social-problems facing the world 

today, is dependent on the ability to support promising: new 
areas- of - science; Consequently, it is-crucial to maintain 
flexibility in the. system even-in a period of level- funding. 


channelled-to-fund promising new activities,- leaving 


researchers in ‘declining'-areas to retire or move into 
other areas-of science-of their own volition. - However; -- 
with a level. budget,.this option is_no- longer available;.if 
Support. for-promising-new areas is to be found,-and found 
prompt ly,- then reductions must-generally first be made else- 


where. These might involve decreasing the suppert for de~ 


clining areas, or for research groups which, although _ 
working in-flourishing research areas, have not succeeded in 


making substantive advances commensurate with the level of 
resources they consume; 


comes to identifying declining areas and Broups. This may. 
be-partly through-lack of experience,-since it is not a task 
scientists have often had to perform in che past. However, 


way. - The: institutional limitations. arise because; although 
there is obviously greatest scope for savings in the case. of 
heavily-funded research activities, these are precisely che 
areas where reduced expenditure is likely to be most.. 


strenuously resisted by senior_scientists-strategically _ 
placed on-science-policy-commitreeg.12 - The social-struc- 


ture of science also limits the use of peer-evaluation for 
identifying research groups or centres which-have- proved 


relatively unsuccessful-in-their previous research. A. 
Scientist asked to judge whether the-funds-of-a-research - 
group in her or his speciality should be reduced, is likely 
to know some of the members of that group personally, per- 
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haps having eallaboraced with chee: in_previous- Fasearchy met 
them regularly at conferences, or-interacted-with-them at 


the-relevant learned society. For that scientist, a. 
decision.as to- whether - or not to-recommend -a-cut-in-funds-is 


exceedingly: difficult to:make,; jeopardizing as it does the 
future livelihood of solienayesr 


that had to be made in more affluent times,. and. one where a 
negative-outcome -merely meant that- new equipment could not 


be purchased or additional researchers could noc be 


Fecruited.-- ‘Gonsequently, when circumstances have forced 


more to do. with soctai or. political faccors than past or 


likely future-scientific performance. 14 - -Furthermore, it 
should be noted that peer-reviewers tend to be recruited 


most heavily from the higher status institutions; the cuts 
they recommerd may cherefore result in a strengthening of 

traditional status hierarchies, when perhaps-what is most. 

needed is 4 redistribution of resources benefiting some of 
those research groups previously less well-endowed. 


To sum up our discussion of the three main problems- facing 
peer-review: first, in a period of level funding, where 


resources fo; cxisting research activities must often be 
reduced if-new areas are te be supported,- peer-evaluation 


seems to: be _proving- increasingly ineffective. as-a. mechanism 


sufficieatly expeditiously;- Because: representation. on. 
scientific-decision-making-bodies reflects-previous -patterns 


of. .-resource distribution, there isa tendency towards the 
uction of that distribution among-both specialties and 


‘Institutions. Secondly, because research in many areas of 


science. demands ever more sophistic ed-centres-in each - 
specialty, it is becoming increasi- difficult co obtain 


the disinterested - - judgements on which peer-review pepedas 


may be inappropriate for identifying declining veakarcb: 
areas or groups. New mechanisms may be required-if scien- 


tific research is to be more effectively and acceptably 
regulated. 
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As economic growth rates in the industrialized countries 
have declined, so-their governments have come to inspect 


prevailing within the basic research community under which. 
Science is-_seen as. the neutral-.and disinterested pursuit of 


to equipment suppliers, new ideas and techniques of -longer~ 


term utility cto industry, as well as cultural contributions 


used to determine the distribution of research funds should 


not be internal ones; decided.upon-and operationalized by 


te h will gradually be pushed 
further and further towards the applied end of the pure- 


One might have expected that, with increasing governmental 
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researchers in each scientific field have been more con~ 


cerned with protecting the interests of that field than 


arriving at-agreement on the relative priorities to be 

given to different. fields-in-deciding between their com _ 
peting claims for public funds. Instead, scientists have 
concentrated on producing reports-that focus on the claims 


of individual fields:!? In the United States, there. has 


Scientific choice, very little attention has been given to 


t to establish priori-~ 


determine research policy will inevitably run the risk of--- 
yielding decisions that can-be-t¢o-easily dismissed (perhaps 


subjecs to regilation by-external. criteria, however, ignores 
the fact that certain criteriz, often implicit, have. shaped 


and-continue > shape scientific decision-making. These 


implicit criteria all too frequently go unrecognized and— 
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the notion that certain areas-of research are more 'funda- 


mental’ than others, or that they are more ‘strategically 
important'. -_ These atr-sften given as justifications for 
the heavy support -of particular research activitiesl8, yet. 
rarely is any serious-consideration- given to precisely what 


is meant by such terms as ‘fundamental' or "strategically 
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Research Councils; 1981/2 


£M 
Agricultural Research Council (ARC) h2.1-- 
(9.72) 
Medical Research Council (MRC) 101.7 — 
(23.32%) 
Natural Environment Research Council (NERC) -54.3-. 
(12: 62) 
Science and Engingering Research Council (SERC) 216.8 
(44.82%) 
Social Science Research Council (SSRC) 20.7: 
oe (4.82) 
TOTAL 435.6 


Source: Appropréation Accounts 1981-2, 
Vol.7, Class X; pp.2-3 (London: HMSO). 


that-certain external criteria helped give rise-to this 


cians and the public at least, that-the discoveries made 
with high-energy physics accelerators would in some way 
help solve the problems involved in producing nuclear 

energy. The scientists: concerned made little effort-to 


disabuse: the public of this. assumed link. Indeed; the 


in Nuclear Science) can only have reinforced this supposed 
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link: No attempt was made to publicize the fact that, 


right from the early 1950s; high-energy physics and 
nuclear~ener gy research bagan to pursue-distinct and: very 


separate. Paths: of development. -.In pounce in the United 


strategic importance over. other basic research. specialties 
in the 1950s and 1960s,-and-which was used to justify: it 


receiving a large fraction of the tocal basic-research 
budget in-Western countries. 0. Later on; political. 


criteria-came to the fore. - When the costs of research in 


this. field. escalated-to-such-a-level that most nations 


could a) Tecra afford .to build and. maintain - national 


acam; + Mehr tsae appealing to many British policiciens 
+i the late 1960s.21 


The problem 1 now is the absence of appropriate iieieliaiit sa. for 


restructuring the. basic research budget — for redistributing 


resources (when this is deemed necessary) away-from certain 


specialties to others. -As we noted eeriters this is fot a 


-2--- 


ossification of tiv peer-review rystem: - Hence, the current 
distribution of tht asic ..search budget-to the different 


areas or specialties: in science is prc ‘nibly more.a reflec~ 
tion of priorities that were © /xed-early in the post-war 


period than-the-result ot < cious planning on the basis of 


explicit criteria in subsequent decades. 


Certainly, there have been some attempts in recent: years to 


Give greate? attention to explicit external criteria. --- 


However, to the outside. observer, these seem-to have been 


— 


mG." concerned wth justifying the existing. distribution of 


resources than seeking to establish future: priorities:on a 


systematic, - explicit, and, as far as possible, impartial 


basis. The Pressures cowards greater public accountability 


h.ve naturally Seen greucest in the most expensive areas of 
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nomic benefits have been considerably greater thgn che “4- 
investments made in these high-cost areas of research. _ 
These studies may-have been. good ‘public relgtions', helping 
to secure the funds needed to continue stich research, but. a 


is of limited relevance unless one also has to hand figures 
on the rates of return for other research activities, 
Moreover, one would also need to introduce into the calcu- 
lation the possible negative effects on-established rival 
firms which are iiot awarded these contracts: _-For such 
firms; the consequence of 4 tompetitor-being given-a lucra- 
tive procurement contract may be a declining market share, 
perhaps ever: threatening their long-term financial-via-- 
bility... Thirdly, and perhaps more importantly, there-is a 
dangers: that goo great an emphasis o. the immediate techno- 


logii:ai and economic spin-off from these forms of research- 
activ-ty will regule in a neglect-of artivities relating to 


off, by virtte, {cr example, of their nied for high-techno- 
logy equipment; may_weii attract aivcution-(and-funds) away 
from those activities where the benefits are more diffuse, 


although no less real: 
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There is also a fourth; more gefieral issue raised by. eco- 
nomic spin-off studies of the sort carried out by ESA. ---- 


This concerns their insensitivity-te the political and ideo- 
logical problems-inherent in any attempt to formulate and 
apply external criteria. Criteria.such as ‘economic 
benefit' are not neutral categories: ‘economic benefit: 
accrues unequally, and by. tio means- everyone wou’i igrce as 
to its value, -particularly-those who have to aiivegt-the 


associated disbenefits. The--benefits are often concen- 


trated on a-relatively small number of firms, yet- spin-off 
calculations rarely consider the question of who bears the 
cost (ultimately the public) and who..benefits. --It- cannot 


necessarily be assumed that-what is-good for a-few firms 
leads to-general social benefit. In most studies 


attempting to apply. external-criteria to evaluate scientific 
activity,-it is only the more readily quantifiable outputs 


that are focussed upon (as-in traditional cost-benefit ere 
analysis). These are invariably economic. in nature rather 
than social or political.- Yet there are.many factors rela- 
ting-te-the impact of different forms-of- scientific activity 
other than just economic growth and-profit, and these are 
systematically ignored in most Stiidies. ---This-is; of 


acceptable to politicians and the media. To take one 
example, in a recent study examining: the economic impact-of 


radio astronomy,23 we found substantial industrial-training 
"benefits' associated with this activity: From the begin~ 
ning, there has been a considerable-flow of trained 
graduates into -high-technology industry, transferring with 


them the skills and techniques-(in-computing, image— 


radio astronomy. Given that-most previous studies in the 
area have-focussed mainly on technological spin-off, this 


pally into sectors-associated with national. séctirity work - 
satellite construction, computer- surveillance, remote 
sensing, and automated finger-print recognition, -foer-.- - 
example. It is thus possible to impose a set of alter- 
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native and largely conflicting values when- applying externat 


eriteria to resource allocation under: which contributions to 


military and: police activities would. be- valued as a ‘dis- 


benefit' rather than a 'benefit', Instead of concluding 


that-radio- astronomy had. important social-benefits because 


of its utility to-industry; the- study could instead-be taken 


as Providing further. évidence of - what are seen-as the. 


insidious links between much basic research and the mili- 


tary-industrial complex. 


This éxaupie- is only intended to highlight a. general ptebled 


associated with- studies which: purport to present an: 'objec- 


tive’ evaluation of external-benefits associated with scien- 


tific research, - This is that they _ignore the fact that in 


modern industrial societies. there exists a plurality of 


values. These reflect differing sets of- political, 


cultural-and economic interests; and are-often in. conflict, 


All too frequently, such assessments fail-to recognize the 


value-laden nature of the assumptions underlying them. -- 


Instead- - they uncritically adopt one particular set of values 


which gives precedence. to - economic considerations over-.-- 


criteria- -relating:- to freedom; ametity, - environment ,. -health, 


and: societal problems. -- Inevitably these studies tend-to 


arrive-at conclusions which. justify.and reinforce: existing 


patterns of resoiirce distribution within scientific. 


research, rather than questioning- whether a radically 


different set of priorities is Perhaps required. 


fa short, the clirrent pattern of red expenditire, - even if it 


is-not explicitly. based on external cciteria, does pre-. 


suppose the existence of certain implicit criteria. - By- 


refusing to accept that external criteria have: any legiti- 


mate role to-play in the regulation of research activity, 


the scientific community avoids crucial questions being put 


concerning-the future direction and use of-seience,: leaving 


the: present structure of- values embodied in the distribution 
of funds to go unexamined and unchallenged. 


REDIRECTING BASIC RESEARCH 


It. is. clear from the above discasstsn that the. d@cision--_ 


making: system in science is. urrently in a state of crisis. 


Not only has-the internal system for-resource-allocation in 


basic research grown increasingly- Yigid, but there exist-no 
adequate-means of linking scientific. endeavour with socially 
determined priorities. Overcoming these problems will be 
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one of the main tasks facing Science in the 1980s. 


Perhaps the major obstacle to change. i is the political and 


ideological resistance of the scientific community; 
channelled through. powerful organizational structures. The 


view put forward by Polanyi chat ‘the soil of academic 
science must be exterritorial in order to secitire its control 


by- scientific opinion'¢ 124 ts still the dominant ethos among 
scientists. - Nevertheless, - the need to restructure and 


redirect scientific activity is now. being: discussed “by. many 


National organizations. The NATO Science. Committee; which 
has assumed the role of -trying-to-promote-change, -argued in 


a-recent ‘Message to the Political, Industrial and Scien- 
sific Authorities of the NATO Member Countries’ that 


Sr agreeing to- break: down baretecs: abolish established 


privileges,.redistribute- facilities, make- international 
rather than national options, . set aside the less prom omising 
and-re-group all-its activities within the framework of a 


more selective international community. ‘2° 


However, sucti. a course impties a considerabie degree af. 
faith in the self-regulation. of the. scientific community; — 


involvement may vell be necessary to structure and 1 eatalpae 


change; -particularly-with regard. to. questions of resource 
allocation, _ The crucial issie here is the-extent -to-which 


it is-possible to integrate. such initiatives with tradi~ 


the: scientific. community. z as-a.whole does not see as threa- 
tetiing. to-their-own-activities.- -For it is-clear that out- 


side participation_in scientific affairs willbe most effec- 
tive-if-accepted by-the scientific community in a spirit of 
enlightened self~interest. 


The question of scientific as apposed | to social control of 


science is, of course, -far from new » and it ig one that 
has generated a great deal of antagonism and -misunder--- 


standing in the past - both within and-outside the scien- 


tific. Community. On the one hand,-scientists have-asser- 


ted; generally with a minimum or justification, that only 
they have the competence to decide on scientific affairs; on 
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the other, there have-been those who have attempted. to treat 


science in the same way as technology, seeking-to ‘pian’ 
scientific research so-as to achieve broad social and eco- 
nomic goals in a way that denies.the-existence of any 


internal logic to scientific progress. 


Our view is that it is now possible to go beyond such. 


apparently irreconcilable positions, and to develop. more 
effective means of regulating science from the point of-view 
of facilitating scientific progress, while-at- the same time 
ensuring-that overall scientific activity is kept broadly-in 


Providing the information-to-enable scientists from other 
disciplines, scientific administrators, politicians and the 
general public to formulate judgements on- the inputs and. 


outputs for each area of-research. In short, science will 
have to become more openly accountable... This will 
inevitably involve reforming and extending the present 


systems for decision-making in science.2/ - This, we 


undermine the basic premises underlying scientists’ 


traditionally voiced desire to exereise-effective control 


First of all, it is clear-that-significant improvements are 


needed-in the data available on the inputs to the different 
areas of scientific activity. At present, few countries 
have a clear idea o% how much is. spent, on.what, and how 
this compares with the- spending patterns of other countries 


taken an important first step in this respect with the -bi- 


annual: publication by. the NSF-of-Setence Indicators, but: 
there have been few matching initiatives elsewhere, despite 
the encouragement of the OECD-Science and Technology: 
Indicators-Unit, which publishes the general statistical. 


digest, Sczence Resources Newsletter. in the absence of 


more reliable. data, particularly disaggregated data; it is 


extremely difficult-to evaluate trands. to set priorities, 
to relate the inputs for different areas of science to the 
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outputs, and hence to arrive at judgements of reiat‘-e pro~ 
ductivity, This-will imply a-much larger statistical - 


effort on the part of most nationsto monizor the distri- 
bution of research resources, but one that is warranted by 


the level of resources currently consumed by scientifi: 
research. 


Secondly, it willbe necessary to overhaul the system by 


which: resources -are allocated-to-different scientists; 
institutions and-scientific specialties. Currently-this is 
based almost entirely on peer-evaluation which, as we have 
seen,-is facing various problems: - Certainiy scientists, by 


virtue of their expertise. and-specialized knowledge, are in 
principle best placed to identify the most timely--and- 


astronomy.) It is-also possible to_envisage alternative, 
more sophisticated methods of peer~evaluation, involvin 


The-possibility of referees being challenged to defend their 
reasons. for rejecting or accepting Proposats may help --- 


alleviate the rather alarming situation documented by Cole 
et al,28 who concluded-from their recent study of peer- 
review in the United States thar, 
‘the -fate of a particular grant application is roughly . 
half-determined by the characteristics of the proposal - and 


the principal investigator,-and about half by apparently. 
random elements which may be characterized as the ‘luck of 
the ceviewer draw''. 
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These various problems with conventional peer-review oe 
mechanisms have been r-nognized in the Netherlands in par- 
ticular,-where some-ot the changes outlined here have 


already been adopted by the Organization for-the Advancement 
of Pure.Research (ZWO).- All-physics-pro»osals are now 


judged-by a jury consisting of physicists from-a range of 
Specialties, together with a number of chemists, mathe- 

maticians, and astronomers: The jury-is-an ad hoc_one,- 
being reformed-anew each year. -It does not meet, thereby _ 


avoiding the danger that its members-may be unduly swayed by 
the. arguments-of-a persuasive advocate for a particular 


merely ask questions and make criticisms of proposed. pro- -- 


jects. - At the same time, expert advice on each-proposai is 
obtained (by mail) from four to six referees. . The - .- - 
referees, at least two of whom are foreign, are not asked to 
recommend whether the proposal should be accepted or 
rejected, but they are asked-to give reasons why they think 
the-proposal-is worthwhile or not. _ The referees’ comments, 
suitably edited by ZWO in order to. preserve anonymity, are 


shown-to the: grant. applications, who havé the right of reply 
to these criticisms and.to those made by the jury... On the 
basis of all this evidence, the jury then grades. the - 
Proposals in terms of four criteria - the ability of the 


¥esearchers, the objectives of the work, the methodology, 
and an overall quality grade. Although the final decision 


as-to which- proposals should be £ 
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formance in-a rigorous and systematic manner. These draw 


upon the methods commonly used by social scientists (for 


example, structured interviews to produce coherent- sets-of 
peer-evaluations, and. attitude-surveys) , and in particular 
the- indicators developed in the 'scien-e of -science’-= ior 
example, publication counts and the frequency with which 


resezrch publications are cited by other scientists. 


Such evaluation methods -are capable of yielding information 
of considerable use to-decision-makers in-that they supple- 


ment and complement the results from traditional peer-review 
procedures. - The tiethods have-been applied toa number cf 
Scientific specialties,32 and have proved successful-in  - 
identifying research centres which, despite being relatively 


generously supported, have failed to. jake scientific -contri- 


interests are such that they prefer not to disclose these _ 
facts and“initiate wider debate in the scientific community. 


ex} “ comparatively large literature on the factors. 
cei'.i ‘ag to-successful technological innovation, there 
is «- sth of systematic information om the conditions for 
Sus..e68 1u-scientific research. _ Certainly, experience _ 
Suggests that the ‘ageing’ of researchers, their restricted 


many ‘Jestern countries. Yet much more needs-to be known 


mobility, -and blocked career-ladders are growing problems in 
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groups, and instrumentational obsolescence, on scientific 
performance. 


In particular, very little is known about the process by 


factors-explaining the trend rowards the concentration of _ 
resources ona few large centtres.. is this-trend inevitable 
and irreversible? - If so, what ate che policy implications, 
and what steps need to be taken-to accommodate them? For 
example, can research councils like Britain's. SERC ade- --- 


centres complete the-task-for which they were originally set 
up, how. can-they be closed or how can their-efforts be 


redirected into new-research fields given that they 
generally-iack the flexibility associated with university 


research while at-the same time they represent powerful 
interest groups acting upon research councils? - It-is clear 


will it be possible to reorient activity to-meet new needs - 
scientific and social - as and when they arise. 
This: brings us to the final point, the urgent requirement to 


Science with future economic and-social needs. As was _ 
notec earlier, little is known about the precise nature of 


with skills relevant to new technologies, such as super- 


conductivity or microelectronics). This is clearly an area 


ripe for-research: -Methods need to be «established for 


all we are advocating here is the replacement of che 
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implicit external criteria: currently responsible for shaping 


the distribution of the-seience budget by a set of explicit 


criteria arrived at as a result of pubiic debate in which-a 


wider range of interested-parties have the Opportunity and 
the informati +r. weaired to parcicipace. 


Again, - institut zonal changes will _be needed in the scien-- - 


prope..ly. applied: In particular,- tore widely based commir- 


tees- -Fepresenting a plurality of opinions, interests -and 
values will be required: The ‘information' provided to - 


help-make scientific decisions will always-be subject to a 


number of different interpretations, - and different -groups- 


with-different values will: identify-and defend-these,; each 


interest. group attaching its.own. political 'weights'-to-the 


various possible-outputs from science. A: heaithy, well-run 
and. open scientific system should have nothing to-fear from 


an extension of the franchise in this way. A system 
dominated. by certain organizational. _interests and. charac-- 


terized by a high level of secrecy_surely has:. We need to 
start. formulating science-policy mechanisms -that- recognize 


the need -for open debate, that admit the inevitabilicy of 


healthy disagreement in a pluralistic system where values 


and objectives often conflict, and that force these values 


and objectives out into the-open..- Implicit assumptions 


underlying a particular science policy (for example, 


assumptions about the relative-importance attached -tc - 


internal criteria compared. with external ones) cowid then no 
ie be hidden from the public gaze. 


interest; would be replaced by greater openness. from ee 
we-would-argue,.a more acceptable-and ultimately more 


effective distribution of scientific resources would result. 


NOTES ; 
1. We. _use. the. term ‘basic. scientific research’ to. refer to 


560). pp. "159-71. The distinctive. we use-later in 
this article is not identical with. Weinberg’ 5,. since one 
of his external criteria (‘scientific merit') is incor- 
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porated in our internal category. -_. 


It must be emphasized chat we aiz talking tere about 


play a central role in decisions on che distritution of 
state funds to applied research - for example, to 

military R&D. -- oe : = 
M. Polanyi, 'The republic of science: its political “nd 
economic: theory’; Minerva t (1962); p.61- 


basic research. Obviousiy, external criteria normally 


- I. Mitroff and D. Chubin, 'Peer review at the NSF: a 


dialectical policy anctysis, Social Studies Of Setence ~ 
(1979), p.200. --- --. 2 “ : : 
This statement: and many of the other observations elow 
are based ofi the-results of interviews by the authors _ 


with over 400 scientists. However, because the inter-_ 


For this-reaton, figures on the percentage of 
scientists holding one view or atiother have not been 

: : re ol ap 
Evidence for this comes from our interviews with radio 


dations on how to spread substaatial-cuts across the 
university system. While the secrecy under which the 
UGC operates makes it unclear what ~érformance- indi- 


cators-(if any) were used, the net effect was to concen- 
trate the cuts mainly on certain newly established - 
universities (most of whom did noc have representatives 
or previous students on this Committee). 
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15. It should be stressed. that we are net advocating chat 


restructtring take place- -ever-right.- Scientists and 


their- support staff cannot be treated simply as dis- 
posable resources,-particularly in the current- eco-omic 


climate. Much more thought clearly needs to be given 
to the question-of policies for restructuring scientific 


activity, inciuding, for example, provision for re- 


training and- greater institutional mebility. 


16. This was a committee of che US National Academy cf 


Sciences sét up-and chaired by George Kistiakowsky 4 


former presidential science adviser: —It had at 


ranging brief - to study and pronownce upon any pro. 


coming under the: heading of scieuce and public policy.- 


For- further-details, see D.S.-Greenberg; The Politics of 
American Setence (Penguin: Books, 1969, PP. 207-10). 


posesbitity of making choices between scientific fields, 


and- therefore arrived at the inevitable conclusion oe 


solutions for- eetincea - growth. Sometimes ete can be 


acquired at little cost; sometimes larger expenditvres 
of funds are needed. ae the cost may not reflect 
the relative value but rather the- need. | _Eaci. area must 


be funded - :according to nese needs aries ® Aprora Panel 


ae emphasis added_- quoted in Greenberg, op: atti; By: 5: 286-7); 
18: fe for esameles | H. Bethe, -‘High- coed physics! (p- 9 


frontier o£ - physics. - -The. discoveries in this field of 


study contribute most to the advance of our fundamental 


understand ng of. nature see: Solid state: theory is still 


> matores! (Original @ isis). == 
19. Some might argue that ~energy physics: although 
expensive, is so imp- . that it must - be included in 
the- 'portfolio' of r: -¢° -‘etivities: supported hy 
highly industrialize ation. he Even if this argument 


is accepted, it does not help in d. ornining the precise 
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But -it -is ise fron /bvious why such discoveries .-.. to 
be made-in 10 years rather than 15, 20 or 25. 
Commenting upon the _setting up of the National - -Institute 


Nature (Vel, 179, 27 April 1957, Pe 835) referred -to high- 


economically- vital- tieid’ (emphasis added), and_-went-on 


to. observe chat "the urgency. of the demands on Britain's 


depleted | reserves of _energy places: the field of. nuclear 


See the-Parliamentary discussiou reported. in Hansard. He 
768, columns 808-50, especially the speech. by P.-Kirk. 


M.K. Evans, The Economic -Impact of ASA R&D Funding - 
(NASA CR 144351, 1976); J-P. Fitoussi-et al.,-Economic 


Benefits of ESA Contracts (esA/C(78) 104; 1978); H. 
Schmied, ‘A study of economic etility from. CERN---- -- -- 


contracts’, IZEr Transactions on Engineering Menagement 


EM24, - - pp. - 125-38. _ _A ~nonqquantitative: but - perhaps _ 


eeerree Se is to be found ia beM. a niveeenaeae 
from-fundamental-science', paper-presented at the NSF 


Workshop on Basic Research and Development for rhe 


Advancement -of Energy-Science--and- Technology (mimeo, 
Fermilab; Batavia, Illinois, March 1982). 
J. ‘Irvine. and B.R. gueee ‘The conomic effects of ‘Big 


28-30. 0 April? 1980: (Ref. ESA SP-151,; ate 1980). 


M. Polanyi (1962), op. ctt., p.67. -- - 


NATO Science Committee (1981); op: cit., p; i: . 
See, for-example, . 0. Bernal-cad-M.- -Polanyi, ‘Ought 


science ‘to be planned?.. Two: opposing views',; Palletin 
Of the-Atomic.Setentiats 5-(1949} pp- 17-20. 


Whether such reforms are possible in the absence of 
wider-politieal-and social changes is a debateable 


Question, but one ws ich we cannot address here. 


in: peer-review’ > Sotence 214 (20 Novewber 1981), p-235,. 
This finding was aptly summed up in the headline to the 
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ew Scientist article about this work, "God does play 


dice - with scientists’ granr3' ew § Setentist, 19 
November . 1981; P. 483). 

See C. Le Pair,- ‘Decision-making on: gran: spplicacious 
in a small country', Sctentia pueeenae ie 6 (1980), pp. 


137-43, for ‘further details.- 


still further the saat of the jury to include 
flon-scientists as well... - 


See B.R. Martin and-J. tevine: ‘Assessing basic 


research: some partial. indicators of scientific srogress 


in radio- astronomy’ » Research Policy 12 (1983), and 


"Internal criteria for. scientific choice: an evaluation 


of the research: in high-energy physics using electron 


accelerators’, Minerva 19.(1981), pp.408-32; and-J. - 


Irvine and B.R. Martin, ‘Assessing basic research: the 
case of the Isaac Newton Telescope’; Socral Studies of 


Science 13 _(1983), pp49-86. 0 
scientific - research and innovarioti can be fourd in M. 
Gipbeas -and- RD, - Johnston, The _Interaction of Setence 


Scientific Policy Gece of Liberal Studies in 
Science, Manchester University, October 1972); 
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CERN: Past performance and future 


prospects 


I. CERN’s position in world high-energy physics 


Ben R. MARTIN. and John IRVINE * 


Science Policy Research Unit, University of Sussex, Brighton BN] ORF, UK 


___I0) a series of three papers, we attempt 10 evalies! «Ne pasi 
scientific performance of the three main particle accz!i/310rs at 


the Geneva-based European Organitation for Nuclear Re- 


have been ibe experimental achievements of the CERN inter- 
secting Storage Rings in world terms? And. third, bow do ibe 
cotputs from the CERN 400 GeV Super Proton Synchroiron 


search (CERN) over the period since 1960, and to aseces the 


fifteen years. ond 
We es in this: paper (paper. i) with the 

Position of the CERN accelerator in world high-energy physics 

relative to those at other large laborstories working in the field. 


We deal peimerily with the period from 1969-10 1978. and 


(ene Hl Marin ond oven nich nt for LEP, the large 


lectroo-positron collider scheduled for completion in the latter 
half of the 1980s. Whai are the consuuction requirements 
(financial and vechnical) associated with LEP, snd bow esaily 


tion arouad the world? Aad, In dhe ight 6? ihe previous record 
of the CERN accelerators, tow har exter is this potential Ukely 
to be realized? The paper conctudes with a discussion of the 


In particular, Ht asks, 


° No.order of seniority implied ( eee ee eee The 
authors_are Fellows of ‘he Science Policy Research Unit, 
where they work on a range of issuts connected with policies 


carrying out this, psenmaeens “une belp 00 freely given 5 
large numbers of high-energy phyticisis. In particular, we 
must thank the senior management at “ERN for allowing us 


fuil_and open access to the laboratcy during our research — 


SPRU, and to the late Sir John Adan, Sir Clifford Butler, 


Professor John Dowell, Dr. Yves Goldschinidi-Clermoat, 
Professor Tom Kibble, Dr. Leon Ledesr.a: %y Cwen Lock, 
and Dr. Paul Musset, for providing use‘: critcal comment 


ott 2 ene — technique sa a's : pao a 


Tiss Caparo nal ipa oe asi eee 
Europeen Suites in nuchear research of a pure scientific and 
fundamental character, and in research essantially related 
thereto. 


On ‘Sepiemba 39 29, 1954, the ‘ anvention for ths 
Establishmen: of a Europe.” Organization ‘for 
Nuclear Research came into operation. This had 
been signed the year before by the representatives 
of twelve sates.’ On the same date, the Conseil 


" Toe lading Sas wae Bagi, Denamait, Povace te 
Federal Republic of Germany, Gresos, aly. i .aher- 
lands, Norway, Sweden, Switrertan#. Be Wag Se oth 


Rese.*: Poksy 13 (1984: 183-210 
Nomtk. ctolland 


and Yogosiavia. Austria has since joisrs wre. 
withdrawn. In addition, Spain was bor we fF 
1960s and repoined in 1983. 


0048-7333 /84/53.00 © 1984, Elsevier Science Publishers B.V. (North-Holland) 
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Exropien pom in Rechwcie Nadiain ~s go 
paratory body set_up in 1952 — ceased to exist, 
although the new organization retained: the 
acronym “CERN.” The main purpose of CERN 
was outlined in Article II of the Conventon, and 
is quoted above. one 


tee papier iar ation irra rptiey 
erators and they usaz? A few years ago, a senior 
CERN official gave the {allowing explicit defini- 


CERN must ia the end be jedged on whether it hat 


By 1954, -the interim Council bad already 


succesded in its primary mission of enabling particle physics 


determined the ‘main- components of the future 


tory seem surprisingly modest. In 1952, for exam- 


‘estimated that the ext stage of development will last seven 
‘Years and cost a total of 27 million dollars after which the 
estimated annual running cosi of the laboratory would be a 


ae) ee eee 


n-year 
arses Blectacany Ar gr epers pail 
of CERN staff was well over 1000, and expendi- 


ture for that year (the first after the completion of 
the PS) was also approximately ten times the early 
estimates - 66 million Swiss francs or about 15 
million dollars. CERN continued to grow rapidity, 
reaching a peak of neatly 3800 employees in 1975. 
It haa since decreased a little in size, so that in 
1982 it employed approximately 3600 staff and 
had an annual budget of over 600 


for the new accelerator, LEP, it is-estimated that 
Phase 1 of this project (to sich an energy of 50 


GeV in tach of two coll:ding beams)-will cost 910 


MSF (nearly 450 million US dollars), while the 
cost of Phase II-(to reach 130 GeV) has yet to be 
publicly specified. = 

In view of this considerable long-term invest- 
ment of scientific and financial resources: in CERN, 
it is pertinent to ask how successful the CERN 
accelerators and the_ high-energy physicists who 
use them have proved. At a prominent theoretical 
high-energy physicist recently remarked in connec- 
tion with CERN, “the tax-payer is entitled to 
know what he is getting for his moncy” 


184 
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with the CERN accelerators rank in world terms? 
At CERN itself, there is little doubi about the 
answer to this question, as the following quota- 
tions mele clear, 


cake uke ae 
The cuspe: is mensured in -sow knowledge ana in ‘raised 


people. On theses criteria, CERN can stand comparison with 
any laboratory in the world ... [4, p. 111}. 


high-eeergy phjnics.on a per with that of any other region of 
the world (5, p. 227). 


ini casa oo besced Gs peeed Ge 
establish just how “great” have been the contribu- 
tions from the CERN accelerators * to the advance 


itis the purpose of this paper to attempt to answer 
such questions. We begin with a brief examination 
of the history of CERN. 


? It should be. stressed that what is being assessed in ihis 
pepe and proer Ii [12) is the ecientific performance of the 
CER acceleras:rs and of the-high~ssergy physicists who 
bave ase them, rather than CERN (ihe laboratory and its 
aap ae —_ 3 

? We have-not-considere’ hare the theoretical outputs from 

from the CERN scoshereicy), suck as their training function. 

For example, the good dengn and rapid success of the 

feces! proton-antiprotos collider experiments (discussed ic: 

paper I [12] clearly. owe much to experience with tne ISR 
where physicists took some time to appreciate the impo-- 
tance of 4~-solid angle detectors. 
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2. The soy of CERN and is mals ser 


smaller 600 MeV synchro-<cyclotron: went ahead, 
the first protons being accelerated in 1957: The 


reaeaEES Guiiee 16 lasom oon 


where the Cost: were too great tu-be borne by most 
individual countries. High-energy physics (or 
nuclear-pbyaics, as it.was ='J] then known) was 


identified a5: one-such field. While accelerators 


similar in energy to the 3 GeV Cosmotron and 6 


GeV Bevatron, then being built at the Brookhaven 
and Lawrence Radiation Laboratories in| the 


as “high-energy physics” (which is now normaly 
defined as “physics done with accelerators able to 
produce primary particles at an energy higher than 
1. GeV" - see, for example, Roche [31, p. 3), and 
because the costs involved have represented only a 


United States, mighs, just be within the reach of 


amall fraction of the CERN annual budget (3 


individual nations (and several accelerators of such 
an ‘energy were eventually built), to go beyond 


these energies was seen to necessitate international 


the years was to result in the construction of a 


series of major particle accelerators. ¢ 
, work began at CERN in-1952 on plans 


percent in 1977 - see CERN [2, p. 25), the 
synchro-cyclotr: will not be further considered 


the point of ew of CERN’s self-confidence - 
before the rival Brookhaven accelerator. Its maxi- 


for the construction of a 10 GeV machine. How- 
ever, after the design-team had: visited Brookha- 
ven, where 2 similar design-study was in progress, 
they concluded that, hy inoxeporstin:, -ne newly 
invented “strong focusing” concept, 11 would be 
possible *o built a 30. GeV ascceleraior for the same 
COT; a6 <12 lower-energy “ weak focusing” machine 
oriyusity planned. The intesim Council agreed to 
this. change, «although with the proviso thai the 


maximum energy be limited to 25 GeV. * This was 
several GeV less than - = -energy of the 


mum energy, at 28 GeV, was also a little higher 
than we CERN Council bad suggested: The PS 
bas_been in operation: ever since, although it has 


ern Europe at the start of the 1960s. By this time; 
accelerators of between 1 and.3 GeV. were already 


fool, aioe certain: element -of — 


existed between the CERN »nd Brookhaven teams 
during the design and construction period for the 
two acteferators, a consicezabe e7d very iruitful 
interchange of ‘ideas_ and_ technical. in/-«mation 
also took place (cf. Johnsen [14, pp. 37-39). - 
Meanwhiis. the design and construction of the 


Operating in Britain at Birmingham University, at 

Saclay and Orsay Laboratories in France, at the 
Frascati Laboratory in Italy, and at "nd Univer- 
sity in Sweden. In addition, various accelerators 
were under design or construction at Rutherford 
Laboratory and, « little later,. at Daresbury 
Laboratory in the UK, at DESY, the German 


and rr at longer te 
initiative, an informal comn. 


Upon this because of their worry. that over-ambition on the 
pat of the design-team might lead them to atcempi too 
Gemanding s project (Interview, 1981). 


1963 to make recommenda’ 

development of high-energy = 

* This was known as the Amst! Cc. ‘ted to 
the CERN Council. For de sila, coc : 
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Thay Somagliine wlich tbeeaieiay Gane) ae 
sume & more permanent bans oz ECFA, the 
European Committee for Future Acculz-stors, re- 
Ported later in 1963. lis two main recon:nerda: 
tionswere thai Europe should build 2 30° GeV 


altogether ‘these would cost up to four times as 
aie es 
Amaldi (1). Not surprisingly, — ze 


than each country attempting to build its own, 1he 
argument that such international facilities in fact 
tequired for their full exploitation a very c.onsider- 
able increase in expeaditure on national facilities, 
must have come as something of a surprise. Five 


years later in 1968, when Britain initially decided . 


agaist participating in the 300 GeV project, the 
Affects of this report were evidently still being felt, 
as this commentator’s observation makes clear: 


‘After successful testing of ibese ideas in 1963 on 


an electren model (CESAR) -‘ouilt at CERN, the 


Intessecting Suwraie- Rings (ISR) project was ap- 


prose" by he = “AN Council i in 1965. cipro 


A rey aa counterps 2 
ee *  4€ CEPN in the Ee. 


ra geomet Diree- 
ig? early years of 


Oe i fear, (oat eonuide able support to 


Seniad or wis 


a. ree vat the | pubsczy « Curae len, ime 
peoud s Sc cee pe wit the soem omen 


Or.6> -hr! at Jean in acme aspects (Weitskan! 13S. fp. 
Vp = 


Dewpae hipecig Ste machine and the for- 
mi¢ztze< -hoical problems that it posed, the ISR 
*.§ 2’, 246d on schedule - the world's: first 
Piotcu-proton interact‘ons in colliding beams 


bein; observed early i: 1971. The ISR -was in 


e-end of 1983, and, like 


luminosity (the- -quantity that; 
when multi iiged ley tia eau sictiae in cr for a 
given process, yields the event rate per second) was 
increased from an initial value of 107 cm~? s-! in 
1971 to 4 x 10” (the design figure) by the end of 
1974, and to 1.2 x 10% by Match 1981. an im- 
Provement of over four orders of magnitude. 
Before going on 1o_ discuss the Super Proton 
Synchrotron, meation should be made of coe ver 
important milestone in the history of CERN — the 


signing in 1967 of an an Agreement with the USSR 


cally, sn ts pe hs eve bon opal redid 8 
p.1197, - 


~ Despite. these question-marks over the wisdom 
of. requesting funding for up to nine additional 
for 


1290), preparations for. the two facilities eed 


Energy under which a join. scientific and technical 
Programme « would ‘be- - mounted at-tbe 10 Gev 


equipment (for-example, » rapid beam-citractor) 
for the Soviet machine, West Europeans physicists 


{6 occup) the summit.of the pyramid went abesd. 


colli tad-cus in tues in sraecting riage hed Bees 


taken up at CERN in 1960, building upon ideas 


developing by D. Ker:t and G. O'Neill in the 


United States and G. Budker in the Soviet Union. 7 


7 However, the first idea of using head-on collisions between 
Particles was apparently due to Wideroe in 1943, an ides be 
subsequently patented in 1943. 


were granted access to ‘what was, for a few years, 
the highest-energy accelerator in the world: §, The 
Agreemen? was extended in 1975. to allow Soviet 
teams to participate in experiments at CERN, and 
again in 1983 to cover new machines at CERN 
and Sexpukhov. 


* [Aa evaluation of ibe ‘past scientific performance of ihe 
Scet_ sopelerator at Serpukhov is made in Inine and 
Martn (11}. 
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bw ioc duals tax a 500 GV Sop Pras 
Syrckroion (SPS). was drawn up in_1964, but its 
Catsiucion was not finally authorized until 1971. 
One factor underlying the delay was the cost of the 
Original design — in 1967, a figure of 1800 MSF 
(about $420 mivkon at the exchange rates then 
prevailing) #us quoted (Amaldi (1, p. 1290)). This 
was. considerably more-than the estimated $240 
millioc- _Saatol an equivalent 10-000 Ge¥ soos 


pleted in 1967 under the direction of R.R. Wilson. 
A second svason for the delay was Britain's initial 
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elects of this significant lead are discussed in 
paper Il (Irvine and Martin [12] which considers 
the relative scientific Outpats of these two mac- 

The SPS operaied eratiesnisly wat mnid-1980 
when-it was closed for a-year to permit construc- 
tion of an associated proton-antiproton ‘collider. 
antiprotons had been put forward in 1576 by C. 
Rubbia, and, following successful tests c' the au- 
cis] ‘lechnique of stochastic beam-coc!ing, (devel- 
Oped_by S. van des_Moer of CERN), che project 
was appro. in 1978. Ii transpired that the SPs, 


withd + +, from the project in 1968.-* Since it was 
intendes| that pre eperetpocr mrne e — 


and in particuls: its vacuum system. had been 90 
well designed that it could be quickly and rels- 


ator, with several strenuously that it should 


arguing p 
be built on: their soil. Eventually in-1970, a com- 


capability. As a-result, it has taken several years 
longer to convert. The main justiZication for both 
these ‘machines was to open up a_new_energy-re- 


Over, 200 ASF of tes taal wiould wow be tard 
from reductions in CERN’s existing commitments. 
Hence, only 900 MSF of “néw. money” would be 
required, half the cost of the onginal design and 


low. enough to encourage Britain to ‘rejoin the 


Cost estimate gave rise to a feeling in the scientific 
community that CERN 


bias ofizn beso 100.presumpivous in its belle! thai Europe 
owes high-energy physics a living [26, p. $93] 


and that 
if it had ‘not been for the shining exainple of Dr. RR 


Wilson’s-bigger but cheaper scoclerator at Batavia, Mlinois, 
the chances are that Europe would still be lumbered with a 


peodiessly expensive roachine (27, P. 1014}. 
_ The SPS proposal: ‘was approved in: 1971, Tour 


gion where several phenom. «2 underlying the cur- 
rently accepted “standard theary” (which unifies 
electromagnetic and weak interactions) were ex- 
pected to occur, and in. particular to Jook for the 
“intermediate vector. bosons” believed to be the 


—— ajOr 
bly a certain clement of 2 “race” between rival 


laboratories around the world to find these new 


the CERN collider, who-early in 1983 anounced 


the discovery of the charged W particle and a few 
months later the nev-ral 2 particle. These dis- 
but they are discussed in paper IL. iis oe 

_Finaily. brief mention should be made of LEP, 
although detailed analysis of the prospects for this 
accelerator are left until paper II] (Martin and 
Irvine -{20}). During the 1970s, CERN. considered 
several options for a major new machine for the 


1980s and 1990s, and, in the end, after extensive 


years after authorization for the very similar mac- 
hine at Batavia. This gap was to be maintained, 
with commissioning of the SPS comin, four years 
after the US accz'erstor became operational. The 


underlying this decision, see Gibbons 7 


sacecas; ee en ae 


ducing beams w.:h an energy initially of 50 GeV 


(Phase I), and eventually . up to 130 GeV. The 
construction of LEP was avthuriz~{ at the end of 


1981, and the first collisions are planned for 1988: 
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This concludes our discussion of the main de- 
velopments in CERN’s history. It should be noted 
thai, during seats ahi the structure of - _ West 


a a varalty acoclerat f acta -of 1-GeV-or 


paysite experimental high-energy physicists, 
now those -experimentalists are largely con- 
centrated in just two centres - CERN and DESY 
{the German facility where international participa- 
tion has greatly increased with" —._ nstruction of 


tron-pasitron ‘colliders Ld a 


Brookhaven,. Fermilab, and Stanford, experim=n- 


en ee) BOS wee we areal ost 


in the-field, with further rationalization a distinc: 
possibility. '! In view of the growing concentration 
of ‘Western Europe's. high-energy physics effort on 
the CERN’ accelerators (nearly. three-quarie ~ 7 
ne Ces a 
on_these for their research), it has obviously v- 
comée.an important. policy issue to devise adeq: 
methods of assessing. the past ‘scientific pers. 


mance of such facilities, particularly in relation _ 
the main competitor accelerators at the three larg: 
United States laboratories. 


pared? In considering this Question, it should be 
noted that hi_a-energy physics is funded primarily 
for scientific reasons — i.¢. for. the contributions it 
is judged likely to make to the advance of know- 
ledge. Experimental high-energy phyrics activity 
does undoubtedly also lead to significant educa- 
tional benefits (in the form of highly-trained scien- 


tific and engineering personnel), to various types 


% See Manin and Irvine [18] for an assessment of the earlier 

-- DESY machines. 

"" See, in particular. the report-of- the recent US Department 
oi Energy Subpane! on Long-itange Planning for High-En- 
ergy Physics, chaired by G. Trilling (34). 


Of technological “spin-off” (ace § 32), anc 
in the case of- mi Organizations: like 
CERN, to the- fostering of international co-opera- 


tion: But the primary reason why Member States 
are prepared to contribute over 600 MSF a year to 
CERN is because of the promise of importent 

scientific results in coming years. Judgement: on 
the likely “.-we performance of CERN will be 
ee ee 


ber-of- (partial) “indicators: of scientific pre: 3) 
(publication counts, ‘citations. numberr of begghe! = 


of “converging partial indicators” in-which a num. 


given field (see Martin and Irvine [19] for details). 
oe ee Oe este cl 
So ference coll ied facilities and their asso- 


ist the scientific ¢ imma radi which can 

~' ombined to give an overall evaluation ¢ of their 
+--- Fibution to “scientific progress.” It_ was ap- 
yx<d in three-different Big Science specialities — 
radio astronomy, optical astronomy, and electron 
high-energy physics (see Martin and Irvine [19]: 
Irvine and: Martin (10); and Martin and Irvine 
[18). In each case, a certain convergence between 
the results based on the various partial indicators 
was obtained, which permitiéd conclusions to be 
drawn as to the relative contributions to scientific 
progress associated with each of the experimental 
facilities. The main elements of the method are 
summarized in table 1. 2 


B To view of ibe comments of acientiats critical diana 


Citation analysis, it is worth ‘re-emphasizing thai citations 
are used-in our method of research evaluation to indicate 


the “impact” of publications on the advance of knowledge. 


But not necessarily their.“ quality” or “importance.” Thus.a 
high quality paper in s stagnating field may contribute little 
to the advance-of ‘knowledge in general. and is likely to 


Tore active field. The.“ importance” of a paper is defined at 
the influence *' would have were scientific communication 
completely free from institutional, social and Pelitical con- 
straints. For example, a potentially infivenual p.. per may go 
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Table) coe 


Main problems with the various partial indicators of eciemific progress anv . Satie Sa Clivcts way be manimized uaing the 

method of converging cistia) indicators 
sid 

based on 


B. Citation 
analysis 


— -. journals. — = - 
2. Verisiioa Of cision rate 
dusing lifetime ofa paper __ 
= unrecognized advances oa 
- th! one hand, and integr.- 
tio of basic ideas on the 
| 
3. Critical citations - 
4. “Hilo-eflect™ citations : 
5. Variutions of citation rave 
with 'ype of paper and 
speciity 


ene, a specialty... - - 
6. Self-ciation and “in Check empirically 234 adam 
house” citation (SC and IHC) results if the incidence of 


Tsay WerEsSAwOD PRK ey OLEDIPE! Keo DE/) | \ 
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cia winl illows, weeps cached o sb 
have been the three principal CERN facilities over 
the-last two decades — the Proton” Synchrotron 


(PS).. ee Storage Rings ISR), and the 


ioe 50 Eedl @ agen CHEN Ge © Sa 


scientific output from the “ERN accelerators, ire 


spective of whether they were used by CERN staff, 
user-groups from_ European Member States,_or 


<iial s paiean i see endings since pecz- 
evaluation, which provides perhaps the least prob- 
lematic of the various partial indicators of scien- 
fific progress, only works well for 2 period of up to 
ten years or so. These are two reasons for this: 
memones tend to fade rather quickly outside this 


eee cognac : tai yan 


“less, 8 


“afi pgs adbgaed th Gor Eieeuamntat. eets- 
fore, is as follows. First, we focus on a: recent 
ten-year period in which peer-evaluation data can 
be used to complemer the full range of other 
indicators. The period chosen was fromm 1969 10 
1978, partly because this is compatible with that 
used in an earlier study. of electron accelerators 


ern Europe. We are_thus comparing the overall 


(Martin and Irvine [18)), and partly because scen- 


contributions 0. scientific_progress made using 


tists iene 10 find it difficult. to assess reliably the 


CERN accelerators. with those made using other 


significance of very recent work. For this 


accelerators. We are wot assessing the performance 
OL CAEN wall (some ct whom, tor oxaninis, wee 


pce eee int com ite tenes 
States, some of the major experimental advances 
(particularly with the ISR) have involved United 
States_ groups. Jt is also important to_ note the 
crucial distinction between (1) the overall scientific 
output from each accelerator, irrespective of the 
scale of funding and number of users; and_(2) 


Statistics. ata fairly high level of a: 


used; for exaimple, the three CERN scceersions 


are treated as-one unit. We-can- then examine the 
degree of -convergence- between: the— quantitative 
indicators based on 


bibliometric (publication and citation)-data, the 
results of which are described in the remainder of 
this paper. The “-'ond_ time-frame considered 
covers the 22 year: « om 196]. to 1982. Given the 
Problem of obtaix:.% systematic poeer-evaluadon 
data over such an <«iended_period, we base the 
assessment largely -1. the bibliometric indicaiors 


their scientific “praductivity” ‘or output per unit 


oe Se: As Oat selene lending 
end numbers of users... tis 


publisbed in 1960, with the result that an assess- 


interpreted with #1 aid of qualitative information 


gregated form, «3 peated ins more diag 


ment sheald, if possible, cover the period from 
then to the pressnt. This requirement, however, 


‘ al) t3e 
Soi~istge tH Salles easeack lat ght bl atharcing 
hit been carried out, and its citation record may reflect 


hw ‘ate Martin and Irvine (19, p. 71] for further Ciscussion). 


poms 
ee 
mn 


be considere’ 2parately, and changes over time 
studied in gre: .- # detail. An attempt is then made 
to interpret ih differences in scientific perfor- 
mance of the TERN accelerators relative to their 
closest coz:|cxitors, and to identify possible fac- 


3 Twenty-five percent of the 182 researchers interviewed dur- 
ing the study recaived their Ph. D. in 1979 or inter. 
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“ Ips to high-saerey plrysics 


below to arrive at ju/igements ( on -the relative scien- 


tific “ productivity” (i.e. scientific outputs per unit 
of inpun) of different-accelerators. 

__For Western Europe and the United States. we 
have drawn heavily upoo comparative data pro- 
duced by C: Roche, Head of the Central Planning 
Office at CERN, and_by.W.-Kirk-of the Stanford 
Linear Accelerator Center (SLAC). ia California. 
Roche has, in turn, used information from censuses 
Conducted by ECFA, the Exsopous Commitice for 


plementary information comes from the laborato- 


ties themselves — for example, CERN Annual Re- 
ports give-data-on numbers of users and annual 
expenditure in previous years - and this has been 
used to check the accuracy of the figures. This 

suggests the Earopear md United 
States figures are accurate t within ° Ne ent or 


‘Data on the 9>,coximaie numr sof 
accelerators 9¢ CERN and DESY >. ui. :, and 
at the three principal U5 laboratories, are shown 
in table 2 for the years 1970, 1974, and 1978. It is 


have been included: in -our-figures.-This has not 


always been the case in previous studies of the size 
of the high-energy physics community, particularly 
in the United States; with the result that some of 
the previously quoted figures have boen rather 


smaller. '* We have then attempted to estimate the 


tas te aah oe eels Os 


the accelerators at each of the { ive largest Western 
laboratories, ‘The figures reven! that, for CERN; 
the figure -bss risen from just under 20 percent in 
1970 to approximately 30 percent in the second 
half of the 1970s, as smaller national aocelerators 
were Closed or diversified from high-energy physics 
1 nuclear: physics and Synchrotron-radiation _re- 
search. It should be noted, however, that the CERN 
figure for 1970 does not include the 100 or more 
European physicists preparing experiments to be 
run on-the ISR in later years - these are part-of 
the-- figure- of 1000 quote! for “other” - West 


3_experimentalists. In-the same way, the 


iiciodar oe tx GA say Wages Hk plow or 
periments for the CERN SPS. Similar considera- 
tions apply to our data on the users of scceleraicss 
at tbe other laboratories. '* Despite these qualifi- 
cations, it can nevertheless be seen thai during the 


19708 the. TERN accelerators. have catered for a 


crea pou tte sanrage (lage, teale Said operation). 
-. The production of reliable data on which to 
base izternational comparisons of funding is even 
here. First, the figures must be fully comparable — 
that is, they must be based on common Categories 
and definitions. In view of the importance of suck 
figures m evaluations of a centre's relative: cosi-et- 


'« In previous papers, we have considered other inputs, such 
as numbers-of technical support staff, as weil as atiempting 
10 seperate capital costs from recurrent expenditure.How- 


here that numbers of weeri and overad fuoding levels (capital 
and recurrent) are = fficient wo define tbe relative scale of 


research activity at each scorlerator {with certain provisos 
-. discussed tater). - 
s For example, see Moravesik [21], p. #4}. We include graduate 


be used in sch comparisons and, a far at posi 
ble, we have adopted their categories. Second, 


must allow for the varying inflation r rates in differ- 
ent countries. Third, fluctuations in exchange ra... 


In addition. the figure shown in ‘ible 2 for SLAC in 1978 is 
emailer than that quoted in_Martin_and Irvine [18, table-1] 
because the latter included a large nur: cr of experiments- 
lists preparing to carry out fulure expenmenis on the new 
eleciron- positron collider, PEP. 
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Table 2 
Numbers of experimental acco physicists (HEP) * — 
—_——— 


(65%) 
Fermilab users m 
SLAC users 200. | 
or 445%) 
Others” 1000 
— Qik) 
Tol US 1500__ 
(32%) 
HEP, in rest of world 700-17007 
(E. Europe, Japan, ev.) ~ estimace * 


ee 4704; ¢ 500) 


-20t 200 
i538) - (45%) 
~ 45014 $50_ 
(1%) (12%; 
(6.5%) (5.5%) 
~ 3007 __ 200__ 
(55%) (t5%) 
1300_ 1290 
(29%) (27%) 
700-1707 700-1700? 
4500 + $00) 4400 + 500) 


sztuenst nun a pce tan eptines nu oad peony 
2 Eermnty Gyn Bet bewn largny compensated for by the gr vth of the subject in other countries such as Japan 


must also be taken into account. Lasily, ibere ix 
that - 


s (bared-on changes in the cost of 
scicntific equipment, salaries, etc.) produced by 
CERN and SLAC"? 10 convert. all expenditure 
figures to 1978 prices:{in Swiss francs for Europe, 
and |_ dollars for the US). Then, i overcome the 
fourth problem, _the. Stanford Research Institute 
“competitive value index” - i.e. the exchange rate 
correcied for ihe teal. purchasing power of the 
dollar in-the United Siates - has been used to 

conv rt all the US figures (in 1978 prices) to Swiss 


"7 Vi has Been awaniod thai. he SLAC infistion index is 
appropriate for the <aber American laboratories and. th. : 
the CERN index is 001 100 dissimilar 10 that for DESY 


192 


ae. For 1978; the. “competitive value. index” 
was $1 = 2.7: Swiss tiancs, an exchange rate that 
both Roxhe [31,.p: 4)-and Kirk (15, The Richter 
Pehl tans HO be teste la:wibte 10 peventor 


estimate o: “tal: world expenditure on high-energy 
physics, which has been used tc convert ihe figures 
tor the major West European and US facilities 
into percentage shares of tota: expenditure. These 
figures. suggest that, in the case of CERN, this 
Percentage has grown from 15 percent in 1970 to 
some 25 percent in the second half of the 1970s. In 
addition, a large fraction of the “other” West 
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European and US funding fo high-eneng> physics (HEP)® = == 7 


W. Europesn funding CERN * 16820) 303 (a0) rT) 365 


in millions of DESY 30 (8) 7 (100) 100 qs) ns 
Swiss francs (MSF) Other - 335 (330; 430 _ (620) 480 6333) -3%5 


US funding ‘10 Brookhaven % 8) 32 8) 3 (45) a 
millions of | Fermilab - t) % (102) 6s (94) 4 
doliars (SM) SLAC 9 2 69) % (38) rf) 


Tebled Basle 


World funding for high-energy physics (1978 Swiss Irancs) oe 
1966 1970 1978 I 


US funding® Brookhaven 145 10 130- 


Oe 650 500 5 
Total US 970__ ns0_. 0- 


Estimated funding‘ - rest of ~ 300-1000? ~ 300-9007 ~ 400-7007 

world (E Europe, Japan, etc.) 

Estimated worid total _ 26BSF(+02) —_—3.0BSF(402) __26BSFIE02) | 

{billions of Swiss francs) (008) (1008) (100%) “= 008) 

* Millions of Swiss franca, = _- : : teas Ween ee 

* A conversion rate of 1$ = 2.7'SF in 1978 has been usd. This is based Boi On ibe official rate, but on the Stanford 
Research Institute “: i te 


United States spend somewhere between 50 ap 


o.. 
ERIC 


188 


i94 BR Hortin end J. irwne / CERN: Past performance and fulwe prospects 


Eurepean funds is spent on supporting university Table _ ae ke 
ours any Approximate cost Per experim-enial high-energy physicist * (in 
1978 Swiss francs) —— aa 


ing the CERN accelerators, a fraction 


funding can be taken into account by: allocating 
the “other” funds in. the ratio of CERN users io 
other European experimentalists (including DESY 
users), having first subtracted a small a te 


Support theoretical work. 
However, such research ia relatively cheap. compared- with 
experimental work (at CERN in 1977. for example, .it 
sccounted for only 1} percent of the total budget - see 2, p. 


25) 0 the effect of this approximation should not be too 
great. 


alos 


suggests that the cost per experimental high-energy 
Physicist does not differ greatly between centres 
(él. Roche j 


Partly to support university users of Brookhaven, 
Fermilab, and SLAC, and partly to “operate the 
Argonne and Comell accelerators in 1978, and the eh 
Berkeley, Cambridge anid Princeton accelerators in -—« ‘OL! 
earlier years. Thus, apportioning part of the US 
other casts to the three national laboratories raises 


costs of PEP and CESR, che 


ts of integrati hese “other” . 
ures for both Europe and the United States 
n the way described are shown in table 5, along- 
side figures for the distribution: of world ‘expen. 


mentalists over the ten-year period 1969-78. The 


_pving a 
uts tr the 


lnputs (or major accelerators at the five main Western high ; se 
ray physics cenures, 1969-78 = See ee be ee 


su ——— __first step in producing these figures was to compile 
® Except in the very early years of the decade, 1969-78. as comprehensive a list as possible of all experi- 
* Except in the inst year Or io of the decade. mental] high-energy physics papers published over 
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by particle physicists, 
which would | generally be regarded as coming un: 
der the caicgory of “experimental _ high-energy 
physics". '* Each paper was then read to establish 
which accelerator (or accelerators 2°) Had been 
used. to produce: th = experimental data reporied. h 


provided by the m main centres. 2? This increased the 
total number of publications by about 4 percent. 
Some of these additional papers involved border- 
line definitions as to whether the results reported 
were. “experimental,” or indeed wheter. they 
should be treated as “high-energy Physics sics” ~ if 

i ubt, included. The 


d when contadecing the figures in table 7; atid 
subsequent tables relating to Soviet and Japanese 
accelerators. 2 


ao ae —— 

© See notes a and b to table 7 for the definitions used. 

% See note ¢ 10 unble 7. : 

2 Citntions to preprints were, however, ipelided in ithe cite. 
tion analysis — bes the nent section for details of ae way 


which they were ureated. 


_ Table 7 presents: the resultant - dau, pving a 
breakdown ‘of world experimental high-energy 
physics. publication: output over the | 
1969-80. 4 As can be seen, ap 
papers were published each year during this period 
@e_ 1000 every two years). Of these, the three 


CERN accelerators were responsible for 26.5 


Over twice as many as the Brookhaven AGS accel- 


erator (21.5: percent), three -times the Fermilab 
figure (8.5 percent), nearly three-and-a-half times 
that of SLAC (8: percent), and some eight times 
the DESY figure (3 percent). - 

Tia put mage Gt ou aa “converging 
partial indicators” is to evaluate the relative scien- 
tific producti. zy (Le: “output per unit of inpui 
the! accelerators, and for this the figures on publi- 
cation. counts need to he set againsi those for 
inpots. The results of this analyzs are shown in 
vable: 8. }t can. be seen that in the case of the 
CERN accelerators, t Percentage of the total 
world output of experimental papers (26.5 per- 
cent) is very similar to their share of world re- 


ap raaiagg ip ema reapray ie 
1969= +5 (sec tables. 2 and: 4), has been matched by 
an equivaler: growth in their share, of the world’s 


igher 
productivity - 11.5_percent of all publicat:ons 
compared with Some 5-7 percent. of funds and 


ich, gives its comparatively late 
field, the period 1973-78 provides 
on of its productivity.In this 


ness also varies over time: the lists used for SLAC, and wa 
lesser extent DESY, for example, were found to be some- 
what i for the late 1960s and early 1970s. 

a For turther discussion of the problems of comparing the 


56-399 O—86——7 


period, its share of publication output (13:5 per- 
cent) is similar io its share of funds and users 
(11-13 abies said we can note thai the 


outputs of Soviet and Western accelerators, see Irvine and 
Martin {1}, 

ad Although this paper is concerned primarily with the ten-year 
period from 1969. to. 2978, some of the work carried..out 
towards the end of this time was not published until 1979 os 
1980. Figures for 1979-80 are therefore included. 
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Table? _ oS eee oe : eS : a ‘ 
Numbers of experimental high-energy phywes s * papers Published in international journals * during the preceding wo years 
1970 1972 1974 1976 1978 1980. Tota) 
a - —_ = — - - 1969-78 
Papersfrom’ = CERN 2354 273 __ 26S 335 320_ 25 1435 
W. European (258) Q7%) (235%) (28.5) (29%) 25.5%) (26.5%) 
acceleration® «=: DESY 3 35 35_ 30 rt © 175_ 
yawn 2.5%) G58) QS) Qs%) {4%) (6.5%) _O8) 
Others - bd 9 105 _ a0 $s ss | 420 
= oe (85%) = * (98) -(98%) (6.5%) (55%) 46.5%) -7.5%) 
Total W. Europe 40_- Ca 40 “ss 435 __ 350_- 2025 __ 
(36%) (39.5%) (353%) = (38%) (85%) (38%) (37.5%) 
Papers from Brookhaven 190_ 190. SS 9s_ oO _- Le 635 
Us ays) (145%) 35%) (8B) 5%) (5.5%) 15%) 
accelerators ° Fermilab - -$ 105 178 180 186 470 
sist _ (3%) (8) 38) (65%) 19%) 858) 
SLAC $5 0 90. 110 105 $s_ aQs_. 
pone G5) (88) as) (9.5%) (9.5%) (6%) (8%) 
‘Others 255- 210. - 180 - 105 9 4s a0 
esas {27%} (WSS) = (155K) 49%) (8%) (38) -133%) 
Tora! US 40 aS $15__ 485_ as__ 330 2370__ 
(51.58) (43.5%) (48%) (418%) (39%) (35.58) (44%) 
iapers irom ber aceaiersiors* 0 Mes 2 US 30 1010 _ 
(Soviet Union, Japan) 29) dys) (19.5%) (21%) 22.5%) (278) (18.5%) 
World total - 945- 1020 1150 1180 111s 930 $410 
(oo: (008) = (100%) (00%) (100%) 00) (008) - 


* High-energy ‘physics is defined as ™ physics done with accelerators able to product primary particles at an energy higher thas 1 
GeV" (Roche [31. p. 3). 
* We have attempted to use the same definition of an * “experimental” high-eoergy physics paper-as the international Particle Date 
Group - that is, the paper musi “contain new (i.e. previously experimental data .... If it is uncertain whether data is 
Rew, it 1s treated as new ,... We exclude theoretical papers. unless s new particle or reeciion property is derived from an analysis (of 
data) which is not based upon any particular phenomenological moded .... Other subjects excluded are insrumenuation, suclear 


laboratories, and dats compiletions such ss Particle Data Group [28) so the final publication list contains a small number of pepers 


published in a variety of other journals. 
All totals have been rounded to the terest S, and all percentages to the nearest Spores 
* All the papers were scanned to establish which accelerator was used to obtain the experimental results. In a small number of cases 


(2.4 percent), more than one accelerator was used. For such cases, each accelersior used was credited with that paper, with tbe result 
that there is a small clemect of “double-counting” in the publication totals. 


SLAC imachines appear io have achieved a rela- 


tively high scientific productivity — 8.0 perceni of F x i oalaee ly 


all papers compared with 5~6 percent of funds Te Gee one 


and users. those from another. The next step, therefore, is to 
However, as we have stressed earlier (see table examine the relative overall impacts of those pub- 


1), analyses based on publication counts alone can lications. 
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Table 8 Oe ee ase UK Ban Lim, 2 
Relative scientific productivity of major accelerators. 1969-78: 
Experimental publications in relation to inpuis 


“3 (% of world toul) 
CERN ~ 25-32" 265 
DESY _ 45° 3 
Brookhaven ~5-7 1s 
Fermilab 11-13" as - 
eae a dacs (133) 
SLAC $66 = a8 


« Exoopt in the last year or s0 of the decade. 
* Average figure for the period 1973-78. 


6 Accelero eat = Serle t tei 
publications 


Analysis of ibe relative naiaicy with: which 


occasionally used Sy experimentalisis, ihe citation 


Counts should (as was the case with the publica- 


tion totals) be around 90-95 percent complete 


(again with the exception of the Soviet and East 


__ One special probiem affecting citation analysis 
in high-energy physics should pezhaps be men- 
tioned. This is the extensive use of references 10 & 
preprint until the paper concemed it published. ?’ 
(This practice ends vii mally at $060 as the paper is 
published, soit normally affects only the citations 
lo @ papet during its first. year.) Such Citations 
have all been credited to the paper that supersedes 


papers from one accelerator (or set of accelerators) 


Is indicatc 
impact on the advance of _wientific knowledge, 
though, ss with other indicators, these are “by nd 


four: years, # ie. the 1972 figures are for the 
number of citations in the 1972 edition of the 
Science Citation Index ™ to al\ papers published 
‘between 1969 and 1972. It cam be Gen thai papers 


means unproblematic. > Three citation indicntors 
~Notal numbers of citations, citations per unit 
input, and citations per paper = are of relevance 
here in assessing general scientific impact, while a 
fourth - numbers of highly cited papers — is weed 
in. the next section to evaluate more specifxally 
which accelerators were respoasible for the main 
breakthroughs in high-energy physics between 1969 
and 1978. - 

- The daia-base from which al, Sour citalion indi- 


Cators were Constructed was compiled by manus!ly 


scanning the Science Citation Index 133) for. the 


years 1961-82. Using this method; full and mia- 
tively accurate citation records were obtained for 


porting eimelts pooroen cs CERN seculereeers 


‘ee Soviet citation wtals must therefore be wrested with 
(peat <ndion ace they may represent an appreciable under- 
wetinate of the “irae” citation totals (see Irvine and Mastin 


PD aes 
® ‘ince Ksest exporimental high-energy physics papers have 8 


tor. Manual scanning largely overcomes the tech- 
nical problems of citation znalysis listed in table 1 
(mis-spelt_ names, incorrect volume or page num- 
bers, etc). And since the Science Citation Index 
covers the 11 main international high-energy 
Physics journals, as well as most of the others 


33 For a discussion of the intrinsic problems in citation analy: 
sis, see Martin and Irvine {19} 


peak cxmiqndate one.to two years after publication. the use 
Ol a pitiod derger than four years does not significantly 
affect the overall citation distribution, though it docs tend 
--- to mask changes over time. — - 
2 While most of the clting articles in, say, ibe 1972 edition of 
the Science Citation Index wil bave_been_officially pub- 
lished in 1972, a few will be dated 1971 even though they 


actually appeared in-1972, t00 late for inclusion in the1971 
edition. Similarly, the 1973 edition will contain a few citing 
arucles dated 1972. We have assumed that these two effects 
approximately cancel out when dealing with large numbers 
of publications. 
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Table 9 
Numbers of citations * io experimental journal articles pubbithed duiing the preceding lout yours 
1972 1974 1976 1978 1980 Toul, 
197241974 
= : - +1 1976+ 1978 
Citaiios io CERN 1429° 2130 __ 1840_ 2080 __ 1760 __ 74% 
papers from haan (27%) Q13%) (24%) (25.5%) (28.5%) (26.5%) 
W. European DESY 2580 190 180 430 1060- 
accelerations = 443%) 23%) - (238) (53%) 415.3%) 44%) 
Others an 320° 4 290 260_ 1680_ 
7 @s) (75%) (5.5%) G.5%) (4%) (6%) 
Tocal W. Europe 2090 -— - 2460 - 2920- 16,230 -— 
(3938) (a2) (328) (438%) (48%) (36.5%) 
Citations to Brookhaven 960_ TH_- TH 4H_ 170_ 2910 __ 
papers from fe (sh) qs) _Q0%) (S$) Q%) (10.5%) 
ur Fermilab 730 1900 2620 13 $320 
soceterators 0.5%) @1s8) - 3%) (28) (21.58) as%) 
SLAC 670__ 6m 1330__ 1230_ 690 920 _ 
a (12.58) (0s) 7%) «s%) (11.5%) (148) 
Others 1130 90 370 370 40 
= = -@158) -(1458%) --€75%) -(%) ~-(7%) (138) 
Total US 2900__ 3190_ 4590 480 2620_ 15,430_ 
(538%) (47%) (59.5%) 59.5%) (43%) (55%) 
420 740 690 $10 530 2350 
-@%) ais) (9%) (6) (9%) (83%) 
World total se. 67m | THO __ 810. 6090__ 28,020__ 
a ("2 (1008) (100%) (WS) (100%) (100%) 


© These figures have been Corived caitig the Science Citation Index: This scans all 11-of the main international journals used. ia 


drawing up the publication’ jet, 2s uli Zs several of the other_journals containing the occasional high-energy physics paper. 
‘Therefore, like Ghe pwblicrtian setets, the'citation totals should be 90-95 pes cent complete, except in the case of Papers from Soviet 
and Japanese accelerators. oe eee ee reeeas ere eee eee re oe ee 
- Bot scanned by the Scien) jembex.) 
» i lve mene sr 10d pg a 05 art 
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earned 26.5 percent of. all world. comteamaal 
high-energy “physics citations for the period 
1969-78. This is almott ‘one-and-a-half times 
that of Fermilab’s share (19 percent), twice the 
SLAC share (14 percent),.two-and-a-half, Gmes 
Brookhaven's share {10.5 percent), and seven times 
probally soem Thus, in terms of abso- 


- ic lapciedl os tocgaies thai varies bb 
the total numbers of citations gained by the accel- 
erators at different centres may reflect differences 
in activity levels as much as the relative signifi- 
cance of their respective experimental outpats. (We 
should rememoer, in particular, that comparison is 
being made between the outputs of one accelerator 


= Aa foied above, only cithiioes in even years are reported 
here. Hovever, the inclusion of citations made in the inter- 
vening oid years doss not lead to appreciably different 
tesults, 
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at Brookhaven and Fermilab, and those of three at 
CERN.) It is essential, therefore, to relate these 


secord, followed by the Fermilab and Brookhaven 
accelerators, with the CERN and DESY machines 
some way behind. _ 

Probably the mast useful of the indicators based 
On total citations is the average number of cita- 
tions per paper. This again takes into account 


per 
input, the SLAC accelerators have had the best 
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Tedis10 eee 
Relative scientific impact ot major accelerators, 1969-78: 


Citauons so relatcon to impets 


7 Gel 
(% of world oul) recent papers. 

= (% of world toa) 

CERN 73-32" %3 

DESY as* a 

Brookhaven 5-7 103 

Fermilab 1-13" 9- 

ae Saks (asy* 

SLAC ——Se6*— 1 = 


Or 
* These figures have been taken from table 5, and rapresent.an 


— average.of the figures for sumbers of users and for ‘voding. 
© Excrpt in the vary'« yeara of the decade, 1969-78. 
© Except in the last year or so of the decade. 


© Average of the totals for 1974, 1976 and 1978. 


difeences i she sale of seaouioes and resend 
activity associated with each accelerator, and -ena- 


impacts of the great mass of experimental pap=rs 
that at best contribut only marginally to scicatific 


progress. Table 11 gives-the relevant data -for 


various fort-year periods between 1969 and 1980; 
the 1972 figures, for exampk, represent the num- 
ber of citations in the 1972 Science Citation Index 
to__experimental _papers published between 
1969-72, divided by the total number of those 
papers. In. terms of this indicator, the CERN 
accelerators again come out behind those at SLAC 
and Fermilab, °' as well as behind these at DESY, 
but ahead overall of the Brookhaven AGS - a 
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that “have completely transformed high-energy 


Physics. In order to focus on such discoveries (and 


Portant, advances), it is useful to examine the date 
on highly cited papers. - 
_- Firs, however, the period 1969-78 needs tobe 


discoveries in high-energy physics had been made 
in the second half of the-1950s and the ‘carly 


1960s, between 196S-and 1968 there were virtually 
none, and from 1969 to 1972 only a few (deep-in- 
clastic scattering at SLAC,» the first indications 
of rising total cross-section in hadron collisions at 
Sexpukhov, and some early results from the ISR at 


was discovered), may be seen a8 an epoch of craly 
“revolutionary” science while the previous period 
from.1965=72 was more one of ‘normal’ ecience 
(cf Kuhn (16). 

.. Dusing this revolutionary period, certain key 
experimental Papers clearly achieved an impact 
many times greater than that of tbe great majors 


Soe Seem. Som the may prove 2 


We pointed out previoudly ihiai, while indica- 


tors based-cn publication counts and towl or 


on the vast sumber of small incremental contribu- 
sions to: scientific progress, they may reveal very 
litte indeed about which accelerators have been. 


responsible for the occasional crucial discoveries 


» ‘The high figure for the Fermilab accelerator is partly due to 
several controversial papers subsequently found io be “ mis- 
taken” - see the following section on highly cited papers. 


scientific knowledge than total numbers of publi- 
cations or Citations. These deta are given in table 
12. sara 

Over the decade 1969-78; approximately $400 


# 32 The caly high-easray pliyscs experiment during the 1960s 
to have subsequently ben sward.S a Nobel Prize was that 


ited. experimental. papers 
of the 1970s concer the discovery of th: 1//pei particle, and 

;, {thst 00 was rewarded by « Nobel Prize._ 
% ‘This work was published in 1969, although the experiment 
had been carried out the previous year.In wha follows, the 
dates given for experiments refer to when the results were 


1980s and 1970s are discussed im more detail in paper I) 
{12}. 
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Table 11 be oot te 7 oe ens 
Average Giations per paper (CPP) for journal articies published during the. precediag four years ieee 
=> == 1972 194 1976 1978 1980 Average jor 
1972, 1974, 
ss oa ae ae ee 197% and 1978 
CPP tor papers __ CERN pei an 30 32 31 32 
from W. Exropes = DESY 40 23 27 $7 a4 38 
ecceternors Others -— 2s 20 ae 20 20 2¢ 
W. Enropean 2 33 29 32 37 si 
= average CPP ans 
CPP for papers Brocthaven ae 23 32 27 16 28 
trom US Fermilab CY) 67 23 32 19 
eccrleraiors SLAC a9 42 WW $8 44 $.6 
Others Ba a3 zo Px) 32 23 
US average CPP 30 33 46 $3 34 49 
CPP for papers froca other 14 19 1s 10 1a La 
eccaleraiors (Soviet Unsoa, 
Japan) : 
World average CPP 27 3a 33 35 30 32 


1 ee ened 08 OR I apes, 908 mans act Serene He Sy pmieent 


experimental jours articles were. published. of 
these, the top 0.2 percent most cited carned 100 or 
more_citations (#2100) in any one year. Ex- 
amination of these.11 papers (see the final column 


‘006 7h Accau be bua: tai Pumiek aconesics 
has yielded ‘most_papérs at these levels, followed 
by the CERN and SLAC accelerators. However, it 
should be noted that at least four of the Fermilab 


of table 12) reveals that they correspond closely 
with | the crucial. discoveries listed above. 
Fur-thermore, it can be seen that the SLAC arcel- 
erators produced over half of these papers (6 out 
of 11), many times more than the accelerators at 


been 
Ce ae 2 Saree sears Delors Gay ware reed 


2 1 is also instructive to examine the distribation 
between accelerant rs of discoveries of a slightly 
lesser. magnitude (the term “major advanc:s” will 


ie Wanboy aelae ben as a strong candi- 
date for a unified theory of electromagnetic anid 
weak ‘interactions. Yet_the Fermilab results ap- 
peared to imply that the simple mode! of Wein- 
berg-Salam. wae insabid and a mor Seeks, 


be used, in distinction from “crucial discoveries”) 


by considering the data on papers cited more than 
30 and more than 30 times in a year. (see the 
saiddle two columns of table 12). These corre- 


spond, respectively, to the top 0.8 percent and top 


2 percent mosi-clied papers over the decade 


» ‘The degree of correspondence is exarained ini cial in paper 
11 (12}. However. it in worth Svessing bere that none of the 
- 11s a “mistaken” 


mm paper. 
* The same conclusion was reached in Martin and Irvine {15}. 
but in 6 much more ientative form. 


wich die dave rates), as- the ‘acoeiog revlon 


Parhape she acer dramenic ie desclopment in neutrino plynice 


Understandably, some of those who laboured long 
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Tab 12 _ oe a 
Numb: Numbers ® of highly cited papers (HCP) tor ihe period 1969-78: Numbers of papers cited a or more times in one year 
oo. Se Bas SS 
HCP for CERN Mie - oa) 1 
W. European = (28%)* 26%) (19.8%) (9%) 
accelerators DESY 2 7 3 0 
- (48%) (63%) (0%) 
Others ta 1 + 
es G.s%) Qs) (9%) 
Total W. Europe 145_ AZ_ 2 
(348) (28%) (18%) 
HCP for US Brookb-wea 7 : on 1_ 
accelerators gh See - (83%) (4.8%) (9%) 
Fermilab 106 ___ AI 1 
(245%) (378) (98) 
SLAC 7s 12° 6 
oho (173%) (24%) (54.5%) 
Others a. ie o- 
(10%) (2%) (G8) 
Toul US 260 vn 8. 
(38%) (5938) (67.88%) (72.5%) 
HCP for “ther accelerators au 4. 2 1 
(5$%) (3.38%) (43% (9%) 
World total 429 | 119” 46. 
(100% )~ - = (200%) ~ (1008) (100%) 


a a 

* ‘These figures have been obtained by scanning the Scien"¢ Cuction Index for the years 1969 10 1982. The likely errors should be 
rather smaller than the 10 percent figure of previous. tables. 

* All percentages have been rounded 10 the nearest 0.$ perceai. 


and hard in constructing new theoretical models to Wei tied Ga Se bs as oa 


elecizo-weak interaction (Coleman (6, p. 122). 
At is far from. simple to decide how such | “mis 


quently described the time fom the first ap- 
pearance of the anomalous results and their even- 
tual refutation as 


g rather than advancing scientific. kniow- 


Violation though, 4 ledge). According to another interpretation, how- 
pence situation pomp lee ia _ ever, the efforts to explain these results were noi in 
ee Ore Seer Te vain: thus, although the Fermilab data Jed iheo- 

. Tn ee eee rists to posit the existence ofa new, fifth type of 
i i quark for reasons ee ee 
they 


“Thr highs eaooab wuacied_a-inesicadoes ance of 


theoretical activity which was just misguided. So they did a nonethsiess served a salable Bearistic purpose in diimabais 
Breat disservice.to the high-energy physics community. They ing speculation about the propertes of particles composed 
held things back by several years” (Interview, 1981). of heavier quarks, properties the upsilon [an important new 
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particle discovered a Witla tnanel ecvaed O01 ic eave (Loner 
man (U7, p. 67}. 


to any assessment o of i impact 


vance Of scientific ata although probably 
not-by as much: as their subsequent citation re- 
cords would suggest. The Fermilab data on highly 
cited papers in table 12_and- subsequent tables 
should therefore be viewed in this light = ~ 

: We can go on from‘here to consider the disiri- 


discoveries {the term * “advances” _will be used). 
The first column of table 12 presex 


continuing major impact over several years, while 
others (most notably “mistaken” papers) ar rela- 
tively short-lived. This effect. can be allowed for by 
analyzing the number of times that. highly Gted 
papers earn more than a ceriain number of Cita- 
tions in a year. The relevant data are given in table 
13. Overall, we.can conclude that the pattern of 


ibe aa that the CERN accelerators produced 111 


such papers, slightly more than the Fermilab acce]: 

erator (106) and significantly more than the SLAC 

(75), Brookhaven (37) and DESY (20) machines. 
ES Set eames ae ae 


How 29. these various, data on highly, cited 


papers relate to the inputs for the various accelera- 
tors? Table 14 presents data on relative acientific 
impact, setting the figures for_n > 15 in table 13 
against the releVant input figures. (Comparisons of 
inputs with data ca more highly cited papers in 


Table 13 - - 
Numba fig papa (CP) tribe pavod 1969-78: Nei Gd papa nd mr nn i 
year* = 
= wis a> ars a>100 
HCP for_--_ CERN 269 3 16 : 
W. European ee 27.5%)" (23.5%) (sse) (5.5%) 
DESY ao 3 
-- (45%) (5%) (3.5%) (0%) 
Others R__ 6 _ + ee 
eee (35%) (25%) (3.5%) (1s) 
Tota) W. Europe “a ce 
G3) G18) (25.5%) 65%) 
HCP for US Brookhaven 3 1 -- 6 
acoslersions (05%) (73%) 7%) (163%) 
Fermilab 232 nn > 
(23.58) (os) (0%) ais) 
SLAC 152_- 31 27 --9-- 
eng asm) (4%) GIS) (s0%) 
Others bd 8 20 r) 
= ests - (8%) (3.5%) 25%) (0%) 
Total US 381_- 134 c) 4_ 
(39%) (645%) (70%) (78%) 
HCP for other accelerators $9_ 10_. 4 1 
(Soviet Union, Japan) (6%) (4%) (45%) (5.5%) 
World wral sea 238 87 18 
qos) qaoos) (100%) (1008) 


© These figures have been obt. ined by scanning the Scieace Cuation Index for the years 1969 to 1982. 


© All percentages have bess rounded 16 the Hearest 0.5 percen®. 
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elative fc impact of major -accelerators. 1969-78: 
Highly cited papers in relation to inputs 
Inpuss*  Numibber'ol times. - 


(% of world total) Papers eamed > 15 


DESY = 45 
Brookhaven ~$-7__ 85 
Fermilab =n? Bs 
SLAC ~ 5-6 19 = 


"These igures have beta uilen Tom able. and represent an 
average of the figuees for numberi of users and-for_tunding. 
* Except in the very early years of the decade, 1969-78. 


* Except in the last year or s0 of the decade. 


both tables 12: and 13 could rote of care 
made.) Again, the SLAC accelerators emerge best 
from this comparison of input-adjusted impact, 


followed d by the Fermilab (but with the reservation 


hines. : 

The ‘output from the CERN: accelerators in 
terms of this itidicator (at-n 5.15) is little different 
from their share of inputs, a Conclusion at Needs 
some qualification if data on very highly Gited 
-_ (4 > 100) are introdiced into the compari- 


& Accelerator « <= Peer-evaliation 


: i of- Various. Big Science 
Specialities have strongly Suggested that, while in- 
dicators based on publication and citation analysis 
Provide essential information on the relative con- 
tributions to. scientific knowledge associated with 
different research facilities, such indicators always 
need to be Considered alongside f 


accelerators included in the present study through 


interviews with 182 * experimental and theoretical 


™ ‘Till does. incinde & sumbes ot ocher iniervices thei 


high-energy physicists: These were carried Oui in 
the latter part‘of 1981 and the first half of 1982. in 


selecting our interview sample, care was taken to 
ensure that as far as was possible the views of 


researchers from Western Europe, Eastern Europe 


CERN, 71 in CERN user-groups in 
Western Europe, 28 at Brookhaven (and in the 


user-group at the State” University of New. York, 


Bulgaria, Finland and Poland. iis 

The interviews were intensive (typically lasting 
14 to 2 hours) and structured, being based on a 
common set of questions but witha a few additional 


Siving brief details of their background and career, 
interviewees were asked to describe their own re- 


tributions of the various collaborations in which 
they had worked, then those from the accelerators 


they had used, and, finally, the overall contribu- 
i from the world's other major accelerator 
ities. In some cases, memories had to be jolied 


or occasionally corrected when discoveries or major 
research programmes were attributed to the wrong 
accelerators: “ Apart from its information value, 
this exercise in systematically recollecting the con- 
tributions of different accelerators was useful in 


as-one unit,“ as was the case with other multi- 
eae laboratories.) In addition_to the five 
major Western laboratories already discussed, we 


included in the peer-ranking three smaller facilities 


» See note 45, below, - = 

- For example. some physicists winogly atiribated ihe Gi- 
covery of ne:tral currents to the CERN SPS rather than the 
Ps. 

- However. ii ahould be re-emphasized that oe are here 
assessing the outPurs from the CERN accelerdiors, not the 
performance of CERN as & centre nor its staff. 
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Electron Accelerator, CEA; and Cornell), two in 


Britain (the Nimrod _and NINA accelerators -at 


Serpu 
hines at Dubna, } Mor and } Yerevan). Thrs all 
proton accelerators with an energy of 7 GeV or 
greater, and all electron accelerators of 4 GeV or 
more, operating between 1969 and 1978, @ were 
included: 
OT the 182 physicists interviewed, relatively few 


were able to rank in full order from 1. (top) to 14 


the accelerators at all fourteen laborato- 


ries. Most regarded at least two or more as having 


made equivalent coatributions, particularly in the 


case of those at-t the lower end of the order. Others 


accelerators and then to rank these in order. Only 
8 percent found it too difficult to undertake the 
ranking. or refused for personal or professional 
reasons. 

. _ Average: rankings on a. scale of 1 to 14 were 
then calculated,“ ‘and the results are given in 
table 15. “ In addition to presenting “overall aver- 
age rankings” for cach research facility (see the 
penultimate column of the table), we have classi- 


“7 Accelerators that only began to yield published results after 
1978 (uch a PETRA at DESY, and CESR at Cornell) have 
.. been excluded. = 


energies were 
excluded. It should be 979d, however, that several of them, 
foe example those at Berkeley, Orsay, Sacley, trascati. and 
Novosibirsk, might well have been placed sbove some of 
those ranked towards the bottom “7 our lisi of fourven had 

-- Usey been, 

Where, tor example: two! “Sbeeleratons. were. ranked first 
qual, they were each given the ranking 1.5 (i-c (1+2)/2); 


where three were placed first equal, they were ranked 2 (i. 


- (142+43)73): and soon. - 
4 It dhould be wtreeced shat these rankings were made prisn- 
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fied the judgements of ihe 168: high-energy physi- 
cists who provided rankings into six groups .in 
order to establish whether interviewees’ Current 


upon q 
interviewed in (1) Brookhaven - 
(2) Fermilab; (2) CERN; (4) re 


and Italy); (6) research teams in two small CERN 
Member States. (the. Netherlands and Norway); 
and_ (6). user-groups of. Soviet accelerators in 
Bulgaria, Finland and Poland... - . __ : 


ures in iable 15. First, a high degree of consistency 
exists betWeen the average rankings for cach of the 
six-gruups; with just two exceptions, all are within 


the scientific con’ 

Dubna and Sesleri accelerators in the Soviet 
Union. Given the problems of scientific communi- 
cation. between East and West; this larger dif- 
ference is probably due to a certain ignorance on 
the part Of many scientists in_the West of the 
experimental work of these accelerators. “ Third. 


a small “self-ranking” effect seems to be evident 


in certain cases; for example, high-energy physi- 
viewed ranked 


sree (who had 200) ranked it 7.6 pone 
Franking”).. However, the effect is not in geticuw., 


CERN scceleratoes in ‘relation to thelr main competitors. 
Similarty, in our previous study of world electron sccelers- 


were mainly aleciron physicitts. (See le Lt. Manin 
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sufficiently large to cast doubt on the relisbility of 
the. resullS. this approach yields.: Fourth, and fi- 
nally, although-a number of senior physicists in- 
terviewed felt that Certain categories of physicists 
- in partbelar: older, established researchers (i.¢. 


“* The difference between te East European rankings of at the 


suggests that at least part of the difference musi be attri. 
buted to an “ignorance” effect. See Irvine and Martin [11} 
for furthe: discussion. 
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Table 15 
Fecr-cvalestion rankings ° of the contributions to high-energy plyeics * made by scnelerstors at} 4 laboratories betwen 1969 and 
Wome 
es by high-energy physicistsin ee eee : average position 
Accelerator(s) Brookhaven Fermilab CERN LargeCERN SmallCERN Bulgaria, "*nkings 
(energy) and Member Member Finland 
Stonybrook. = ‘S:ates States Poland 
(Sample size) (26) Q3) ° (39) (45) @1) as) (168) ——_ 
_ Proton machines = es oe : me = 7 
Argonne ZOS 69 - 66. BS 16. 16. _ 7% 1.6 6 
_—2GY (£03) (402) = (#03) (+03) (0.4) (405) (20.1) 
Brookhaven AGS 31 41 39 37 MM 36 37 ‘4 
_ 3 none (#02) (202) (402) (+02) (+03) (403) (+0.1) 
37 2300-26 223 20 24 2 
(401) (40.1) (401) (+02) (+02) +0.1) 
322 Ws 1a : 110 10.3 115 12 
(0.2) (402) (402) (£03) (20.5) (201) 
32 34 3.6 30 ou 3 
(402) (40.1) (#011) (+02) (403) = (+01) 2 
12.0 122.0 «321. 418 122 2 le 
(401) (£02) (+02) (+03) (#93) 9 (#01) aes 
94 8.9 86 99 10.1 9.1 9a! 
(£03) (202) (+03) (0.3) (£05) (40.1) 
86 91 89 78. 2 82 Jat 
(£03) = (40.3) (+03) (40.5) (40.6) (40.2) 
CEAG@6GeV.. -_ -. 90 a7 9.0 94 10.1 12 92 gat 
_ Li) BYPASS (+3 oe (203) (403) (+03) (£03) (404) (+0.1) - 
pred) = 279, _81. 89 86 9.1. 8 l=¢ 
(Q2-GeV) ree (#03) 9 (40.3) (40.3) (£05) (£05) (#01) 
Daresbury NIN. 96 10.7 9S 97 10.5 10.7 10.0060 Oo 
(4GeV) (40.3) (£03) = (#03) (202) (403) (405) (401) 
DESY (i) (7 GeV) | 63 _ $0 $0 $7 49. 340 | Sa 3 
--(i DORIS (5 +5 GeV) (+04) (#02) (402) (40.3) (403) (403) (+01) : 
SLAC (i) (20 GeV). — 13. a i - 13 Al -14 16 13 1 
_(U)SPEAR (444 GeV) (£0.1) (203) (201) (201) (#02) (402) (401) | 
Yerevan 121 1250 123123, 421 M9 12200 be 
_ 6Gev) (£02) (40.3) ed (£02) (40.3) (#04) (#0) 


“1 1 = highest ranking: 14 = lowest ranking = - 

> Clune wo ter afar of era sch ot neat pc or ocho ca iG imal SE 3 

* The figure in brackets indicate the root-mean-square between the assessments made by the high-energy physicists, Gving 
~ tome approximate idea of the divergence of opinion within the different groups. 

* Ditferencs of only 0.1 or 0.2 in the overall average rankings are not statically significant 
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those who have been active for 15 or 20 years), 
theorists (who are generally less “committed” to 
particular accelerator. centres), and expérimenta- 
lists who had used a large number of accelerators 
>would be likely to give more- balanced and 
reliable rankings, we found little eviderice that this 
was the case. In this respect, table 16 (final two 
columns) considers the rankings made by those 


situa these PAID shagpade baleen ed che aes 
while table. 17_presents ‘data on the judgements 
made by theorists as well as. figures “weighting” 
individual physicists’ rankings by the number of 
accelerators used. As can be seen, these groups of 
researchers hold relatively similar views on acceler- 
ator performance to those of high-energy physi- 
cists in general. 
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Aeccleraiorl Koceleraior(s) Seif-ranking Peer-ranking PhD. awarded Ph.D. alter 


(energy) beforelor__ - W96Slorno - 
en during) 1965 (» = 89) Ph. D.)(n = 79) 
. Proton machines OS. See Seer ee Be, Gh 
Argonne ZGS _ 7 £05) 7.40.1) 7.% 40.2) 7.20.2) 
(2G) - (#12) (9 =156) 
Brookhaven AGS 3.4 +02) 3.%(£0.1) 3.4 +0.1) 


- 33GeV) | (n= 47) (#121) 


2.X 20.1) 2620.1) 2.20.1) 2420. 
(7-108) (n= 6) 
Dubos 11.4204) NS201) 11.0.2) 11.4 20.2) 
—410.Ge¥) (n= 9)- (42159) 7 
Femilab __ -26( $0.1) 3201) 3.1(20.1) 3.4201) 
__ (400 GeV) _ (n= 43) (4 #123), y gnteates ea 
Moscow ITEP - 12.%26.1) 12.2% 20.1) 12.%20.1) 
- (7 GeV)... ae (4 = 168). -- oo: -- ote 
Rutherford Nimrod 7H20.4) -9.4 0.1) 8.20.2) 9.4 $0.2) 
_(7 GeV) (a= 23) (a= 145) _ see, 3 Shed tees 
6.22 0.4) 9.0 +0.2) 8.20.2) 8.7¢+0.2) 
(70 Gev) (ne 19) (9 149) 
CEA G6 GeV _. - 90.1) 9.% 20.2) 9:1( 20.2) 
4G) BYPASS G43 GeV) Z (a 2165) ener ee 
Corpeil - 8H 20.1) 8.4( + 0.2) 8. +02) 
— (12 GeV) -— (n 165). sie ee fala we 
Daresbury NINA - 30.4% 20.1) 9.% 20.2) 10.0( 4 0.2) 
_4GeV) (n= 163) __ eee ae 
DESY (i): GeV Sitzeay $.4(+0.1) $20.2) 5.% +02) 
_ (ii) DORIS (3+ 3 GeV) (a2i%) (a = 151). Jkee Sie mee ee 
SLAC()20GeV _1.3(40.1) /1.340.1) 1.201) 1.2% 40.1) 
(Gi) SPEAR (444 GeV) (n217) (#2152) oe Wee ie dae. Mae 
2 - 12% 40.1) 12.(40.1) 12. 20.1) 
oe > _ . in = 168) Z ae 


giving some approximate ides of the divergence of opinion within the different groups. 


prgeensrein 
® The sample size of each group is denoted by n. 


“One of. ie main diicoiinns with acide. 
Gah ic sol Goutted. the pecklics ‘of ove codes by our peer-evalul 
researchers’ natural worries about the implications tain degree of confidence can be placed in. the 
for themselves and colleagues of negative judge- Tesults. This said, we are now in a position to 
ments, : particularly pas ies Of financial pres- summarize the peer-review findings. The final col- 
sure .on funding bodies. “’ However, given the umn 0/7 table 15 converts the overall average rank- 
comparatively small systematic variations found in ings made_by physicists. back into the relative 


the rankings made by the very different groups of 

high-energy physicists, it would seem [fair to con- 

© See irvine and Maria (13) for & discussion ot ihe problema 
that now confront peer-review as 4 mechanism for scientific 
management. 


“208 


positions of the accelerators at the fourteen 
laboratories. The clear conclusion here is that the 
SLAC accelerators are regarded as having contnib- 
uted most to the advance of high-energy physics 
over the period 1969-78 (see note a to table 15), 
followed by the accelerators at CERN, Fermilab, 
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Table 17 


8.204) 
3.4 £0.2) 
CER Gi) PS (38 Gevy. 2.3% £0.) 
(ii) ISR (31 + 31.GeV) 
11.1(204) 
3.4 £0.2} 
11.4% 0.3) 
9m £04) 
8.1(+0.5) 
CEA(6GVY 9.( £03) 
(it) BYPASS (3 + 3 Gev) Beh 
Carnell. - 8.% 40.4) 
02Gb nap an bide we 
Daresbury NINA 10.1( + 0.3) 
(4 Gev) ee 
DESY.(i)7GeV _ _ _. 5.5(+0.4) 
_.(ii) DORIS (5 + 5 Gev) € cupeatces 
SLAC(i)20GeV 1.40.1) 
-. i SPEAR (4+ 4 Gev) cae 
Yerevan 11% +0.3) 
~ qe Oe 


* 1 © highest ranking; 146 © 


high-energy 
- * ‘The ample size of each group is denoted by n. 
© Root-mean-square variation of less than 0.05. 


Brookhaven and DESY. in that order. As ‘stated 
earlier_(note.45).. these findings. are closely in line 
with the opinions of ‘electron high-energy physi- 
GiSts reported in Martin and Irvine [18). 


9 Keceleraior outputs = an overall assessment of 
the period 1969-78 


a 


__In concluding our assessment, it is “necessary to 
consider the overall picture of the relative scien- 


Rankings of accelerators * for 1969 to 1978: : Views of theonsis and rankings weighted by number umber of accelsraiors used > 


Rankings weighted Overall 
by sumber of - unweighted 
accelerators used sverage 
rankings 
(n= 168) 
1.40.1} . 1.60.1) 
3.M£01) 340.1) 
aac tw 203) 
1. £01) 1.4 £0.) 
aac 21204) 
12.4(4:0.1) 12 £01) 
8.92011) $i 20.) 
2.4201) 8.% £02) 
9X01) 92% 20:1) 
8.0.1) 8.40.1) 
9.401) 1000 0:1) 
$.X£0.1) 5.4 +0.1) 
13° 1.Mz0.) 
124 +0.1) 12.4 £0.1) 


Jowes: ranking The figures in bvackets indicate the root-mean-square variations between ihe assessments 
lysicists, giving some approximatc idea of the divergence of upinion within the different groups. 


tific perlormance between 1969 and 1978 of the 
CERN accelerators and their users. that is- pro- 


vided by the different partial indicators taken to- 


tors, for example, have ‘been responsible for the 
greatest number ¢ of publications. and citations, 1 and, 


behind the Fermilab, SLAC, arid DESY inachines, 
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in fourth position for the ten years under consider- 
: ‘Tic preevalaniice seals tcpeseal the stews Fermilab should be ranked ahead of Brookhaven, 


Of scieniists who, in according relative positions to but behind CERN, agreeing with the relative posi- 
laboratories, have tried to balance small-numbers tions _sccorded to these three by the peer-re- 
of crucial discoveries (for which the number of viewers." 


Papers cited 100 or more times in a year provides a ln the light of these results, certain comments 


useful indicators) against overwhelmingly greater »-ould perhaps be made about the notion of “con- 
numbers_of lower-level contributions (for: which vergence” between partial indicators thal | was 
Gitation totals provide 2 reasonable indication of - introduced in our earlier work on assessing large 
overall impact) “ Thus, it is notable that, while basic reacarch facilities (particalarty in Martin and 
the_CERN. accelerators rank highest in terms of Irvine [19]. It is Gear that, in the present study, 
both total citations and “advances” (papers citec the indicators do not all converge, and the set of 
15 or. more times in a year), the physicists inter- indicators used needs to be interpreted with some 


viewed nevertheless ranked them second for the care. In part, this lack of overall convergence is 


eS ine ist iat noe at ee 


whether or not-accelerators have been responsible 


for the “crucial discoveries” that transformed the “physi 

field. In tine with the figures on very highly-cited poe pret ape oy Cian 
papers_(ni >-100) showing SLAC as having been tion and citation totals lose much of their utility 
responsible for over half such crucial discoveries, since they tend not to reveal where the crucial 
they ranked the Stanford accelerators ahead of discoveries_were made Citations per paper can 
those at CERN. In addition, among the five [acili- also be. misleading ‘unless. great. care.is taken.1o 
ties, only. DESY. apparently failed to produce a _—=ideailify, and allow for, the rather greater numbers 
single such_crucial discovery during the period of “mistaken” papers that appear to surface dur- 
peels ncemed | Brookhaven and Fermilab felis ing such periods of theoretical uncertainty. Even 

Brook! 2 


© The DESY imechines ware, however, rempensitle for weve 
oval fst nomathaan ajc vance: the acovery of 


ing the output of the world". six largest proton-accelerators. tecibties than Fermilab and Brookhaven 
Table 18 Ba ages a 
Accelerator output, 1969-78: Summary of the relative positions suasasied ky the various partial indicaion 
suited, os Publications Toul Gitstions “Advances” ° “Crucial Pose 
aes citations per paper discoveries” * evaluation 
CERN 2 2 4 2 3- 2 
DESY _ 3 3 3 3 3_ 5 
Brookhaven 2 4 5 4 j= 4 
Fermilab 3 2 1 2 j= 3 
SLAC 4 3 Seo Sse -41 1 - 
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completely _ sati factory. since “slightly _ different 
answers are obtained according to whether one 
focuses. on. “crucial _ discoveries’; ** major 
advances.” or “advances.” For a subject in the 
throes Of revolutionary change, the first of these 


the CERN Scissor 
high-energy physics facilities over the period 
1969-78. It_ seems clear that, over this period, 
users of the CERN accelerators contributed less to 
the advance of scientific knowledge than those_of 
SLAC, but more.than those.of any other experi- 


yields: results that are ‘most Consist ent with peer- 


mental centre. However, if we. take into account 


the differing Jevels of resources invesied in each 
accelerator oray (see tables. 8. and. 10), then 


the ten years in question than those of CERN.-? 


In paper H [12], we look more closely at the 


of 
Scientific. outputs on the basis of a single 
dibliometric indicator (for example, publication or 
citation totals). To attempt to do so may lead to 
results that are seriously misleading. Second. even 
when. ‘Severa eral indicators me. used together, ner, nie 


under consideration, information that is accessible 
to “outsiders.” such as policy-researchers, only 
through intensive interviewing. of_the type de- 
is amir to obtain 


s¢ribed earlier. Even if i 
quantitative ——— 


various sais associated. with each of the biblio- 


ing: partiat indicators can be_used_ =-adeiuedly 
only with great care — to yield what appear to be 
reasonable results comparing the overall output of 


3) As a result, we would argue that there is little poirt 1 
attempting to calculate correlation coefficients between the 
tesults_based_on different indicators - this would be more 
misleading than useful because it might suggest spurious 
Precision. 


atthe changes in their performance over time, 
especially in the period since 1978, and at the 
factors explaining theirs. pe: formance ‘relative ‘to 
that of other major acceleralor: around the world. 


performance of the individual CERN accelerators, 


N) E._Aoaldi. CE od the Big Machine, Notwe 214 
~ (1967) 1290-92" 9 
(2) CERN Annual Report 1977 (CERN. Geneva, 1977): 
(3) CERN Annual Report 1980 (CERN. Geneva, 1980... - 
(4) Meeting on Technology Arising from High-Energy Physics, 
CERN Courier 14 (1974) 111. : 
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2? in paper II, we shall sce how the situation has changed very 
significantly during the early 1980s, particularly as a result 
of the discoveries at CERN of the two Intermediate Vector 
Bosons. 
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Il. The scientific performance of the CERN accelerators 


John IRViiE and Ben R: MARTIN * 


Scrence Policy Research Unt. University of Suisex. Brghion BN] 9RF. UK 


Final vernon received February 1984 


- In a series of three papers, %, we atiempt to evaluate aluate the past 


the Geneva-based European Organization for Nuclear Re- 
search (CERN) over_the period. since 1960,.and to assess the 
future prospects for CERN and its users during the next ten to 
fifteen years ~ 

We conctmed ourselves in a previous paper (Paper 1 -~ 
Marun and Irvine (51) with the position of the CERN accelera- 
tors 18 world high-energy physics relative to those al other large 
jorses working in the field. We dealt primarity with the 
Period from 1969-10 1978, and attempted to establish how the 
expenmental output from the three principal CERN. sccelera- 
tors. taken'as @ whole, compared with ‘that Irom other major 


é partial. 
Previous siudies of three Bug Science specialties. 
: - 1m contrast, this paper (Paper 11) focuses in detail on the 
scientific: performance of each of the CERN accelerators taken 
taduidually. In particular, it asks, first, how the outputs from 
the CERN 28 GeV (Bigs or billion electron-volis) Proton 
Synchrotron compare with those from a very si 33 Gev 
American accelerator at Brookhaven National Laboratory over 
the past two decades? Second. how great have been the experi- 


* No order of seniority implied (rousing first authorship), The 
authors are Fellows of the Scence Policy Research Unit. 
where they work on-a range of issues connected with policies 
Tor basic and.applied research, They graistully acknowledge ae 


carrying out this research, and-the help so freely yien-by 
large numbers of high-energy physicists Thanks are also due 
to_a number of colleagues at SPRU, and to the late Sir John 
Adams, Sit Clifford Butler, Professor John Dowell, Dr. Yves 
Goldschmidt-Clermont. Professor Tom. Kibble, Dr.- Leon 
Lederman, Dr. Owen Lock, and Dr. Paul Musset for provid- 
ing useful cntical comment _on earlier drafis of this paper, 
However, the conclusions remain the responsibility of the 


authors alone, 


Research. Policy 13 (1984) 247-284 
North-Holland 


mental achevernents of the Intersecting Storage Rings in world 
terms? And. third, how do the outputs from the CERN 400 
GeV Super Proton Synchrotron ‘and from a riva) American 


machine at Fermi National Acceserator Laboratory compare? 


Attempu are then made to identify. the main factors responsi- 


ble for determining the relative scientific performance ¢ f cach 


CERN machine. — -- © pian 


These factors are of relevance to the subject of 01 third paper 
(Paper Ili ~ Manin and Irvine (52) which sets-out 10 assess tbe 
Suture prospects for CERN and in particular for LEP. ihe large 
electron~positroa collider scheduled for completion in the lanier 
Part of- the -1980s.-What- are the construction fequirements 
(financial and technical) associaied with LEP, end how easily 
will Uney be met? How dots the scientific potential of LEP 
compare-with that of other sccelerators under consiruc- 
Gon around. the world? And. in.the light of the previous record 
of the CERN accelerators, 19 what extent is this poiential Likely 
to be realized? The paper concludes with a discussion of the 
extent to which predicuve wes cas be utilized: in the 
formulation of scientific priorities, and of the problems in 


Ivvine {51)). the. methodology_used to 5 evaluaie the 
past performance of the CERN acceleraiors, and 
obtained an overall picture of their position in 
world high-energy physics, we now turn t 
more detail on the scientific oulputs of the indi 
ual CERN machines. ' The time-horizon taken in 


"asi Paper 1 151}. it must be stressed that what i is s being 
ass ssed here is the scientific performance of the CERN 
eccelerators and those who have used them. rather than 


CERN per se. 


0048-7333/84/53. 00 © 1984, Elsevier Science Publishers B. v. (North- Halland) 
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this. paper is the period _ from 19617 to. 1983. 


However, given that this spans a:number of quali- 


CERN accelerator has proved: Finally. using the 
results of interviews with a large number of high- 
energy physicists in North America, and Eastern 
and. Western Europe. we shall discuss in detail the 
faccors that appear to have determined the relative 
scientific Contributions of the CERN accelerators 
and how these have changed over time. 


2 1961 to 1964: ‘The commissioning and early work 


at- CERN- The smal} 600-MeV-(million electron- 
volts) synchro-cyclotron: had been: operating -for 
three years, and had already claimed a notable 
discovery (the electron decay-mode of the charged 
pi-meson). It had also been used to carry out a 
Pioneering “g-2" experiment (to. measure the 
anomalous magnetic momeni of the muon). the 
first in. a series of ever more precise experiments 
Spanning two. decades for which CERN was to 
become famous within theworld high-energy 
physics community. Moreover. the race to com- 
plete the construction of the 28 GeV Proton Syn- 
chrotron ™ ahead of the rival 33 GeV Alternat- 
i (AGS) at Brookhaven 
al Lat ratory on Long Island.. New York. 
had been won by several months-? However, de- 
7 The very first_papers from the. CERN” PS were in fact 
published in 1960 However, they were so few in number 
t sion would make no difference to the-resulis 
teporied below. We. chose 10 exclude them because no 
citation data are available in the Science Citation Index [62] 
for 1960. 
One of the main reasons for this was that Brookhaven built 
an_electron analogue to test the new alternating-gradient 
technique. The CERN team appatently lacked the staff and 
funds-needed for this: However. because of the close links 


that had been forged with Brookhaven. they were neverthe- 
learn” almost_as much from the Amencan 
analugue as they would have from their own facility. Conse- 
quently. CERN was able to concentrate its efforts more 
iramediately on the construction of the PS. giving it several 
months’ head-start over Brookhaven. @ lead which 1 main- 
tained right until the accelerator was commussioned. 


vn 
war 
sl 


put Trovti ihe PS. during. the period. 1961 io 1964 
was rather limited. as. we shall see nélow. Perhaps 
nN was_that. although the first 
experimental run the PS took place in 1960. 
overall the preparations for experiments were gen- 
erally still at a very early stage. Thus. while the PS 
was capable of accelerating particles and prose 
ing a beam, it was - 
Not yet ready to particpate fully rT ‘resaigh: Ths. as whs 
the first nch harvest in regard to these new quesuions was 
not reaped here. This is why the first qui im states, the 
firs new panucles, were in fact Hot discoverrd here (Weiss- 
kopf [73. p. 13). 


_The full effects of not being ready 10 mount a 
comprehensive experimental programme can be 
seen. in table 1. Certainly the PS began to yield a 
€ number of reseatch papers relatively quickly. 
accounting for.10 Percent Of the total world Output 
of journal articles in the field during 1961-62, and 
nearly 30 percent in 1963-64. This was not.only 
m 


tor at Berkeley in- Californi 38-percent and:23 
Percent). However, the CERN ‘papers reported 
data from relatively simple ‘experiments, and con- 
sequently their. overall impact appears to have 
been considerably less than that of publications 
from the other front-line proton accelerators. Ta- 
ble 1 shows that: PS publications received 14.5 
percent of all citations made in 1964 16 experimen- 
tal high-energy physics papers ‘published between 
1961 and 1964 — appreciably. less than the AGS 
fee percent) and the Bevatron (35 percent). The 


new accelerator) and that of 3 
(not unreasonable for an accel 
Opérating for ten years). In addition; in terms of 
highly cited papers. the Bevatron (with 33 papers 
Cited 15 Or_more ‘times. and 5 cited_30 or more) 


and the AGS (with corresponding. fi ures of 24 


successful than the CERN PS (with figures of 9 
and 1, respectively). In his foreword to the CERN 
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Table 1 oa 


os of Citaucns*  Avsage  Highh cued papeo 
expenmental (0 wk of citations = (Number cuted a itmes) 
Papers” - past four per paper 


ROS) 373%) 193) (15.0%) (120%) 
Brookhaven AGS (33 GeV) 15. 0. o-- a0 M$ 4 4. 
(4.5%) (1.0%) 023.0%) (270%) (29.5%) (28.0%) (100%) 
Brookhaven Cosmotron(3 GeV) 45 | 38 9 ys 
az $9) (SOF) (11 0) 00%) = (23.5%) (25.0%) 
Berkeley Bevatron (6 GeV) MS 1000; 34 Me sl. 2 0 
a (32.0%) (23.0%) (35 OF) : (37.5%) (293%) (50.0%) : 
Other US 2B SO-- 210 - 3.0 oO to 8 0 
(6.5%) 190%) 185%) Be 00%) 46.0%) 2 : 
Towal OS 230285 2000 | 4] Bo. 1-- re 1 
(61.5%) (480%) (77.5%) (85.0%) —(8R.OF) (100%) (100%) 
Dubna (10 GeV 7% nn, ry) 0 0 0 6 
Se eaten (183%) 20%) (3.0%) . 2 . ce 
Moscow ITEP (7 GeV) US: Wi. 10... 05 0 0 0 0 
ae = (2.0%) (3.0%) (0.S%) 
Other - rest of world - - te - - - - - 
Total ~ ett at world 78 © os 0 0 é 0 
(203%) (15.0%) ¢ 
World tonal 395. $35 - 2590 28 Re 
= Ta 1008) (008). (100%) 100%) (100%) 1008) 008) _ 


a 


L run and Irvine {Sl} alsa apply 
res have: been rounded to the nearest 5. 
‘an sutancn_Ggures have been rounded.to the nearest 10. 
© All percentage figures have been rounded io the nearest 0.5% 


aaa Report 1965, J.H, Bannier. then President 
of the CERN Council; referred to ¢ in decisions 


which y unexpected. 
ane The-data on highly c cited papers: suggest, that most 
will enable CERN. 10 Maintain its Position as one of the 
wo:id's three leading laboratories in the field (Bannier (4. p. of the discoveries in this Particular energy range 
3b. were made not by the PS but by the Brookhaven 
AGS, as the CERN Director-General himself 
and. these various bibliometric aacsion would admitted at the time: ; 


Suggest that, at the end of 1964, the CERN PS did 


indeed rank in third position. signficantly behind aged bed results cate been. mate smpreins 
he Some. Fecemt sensat discoveries were 
the Berkeley Bevatron and the ¢ Brookhaven AGS _ elaewhese (Weisakopt (73. p. 18), 


accelerators. See _ 


In opening up a new energy Tange in high-en- Three discoveries at the AGS stand out in particu- 
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Ww the discovery that there are two different 
types of neutrinos;. - 
(2) the discovery of the omega. minds as particle: 

i3) the discovery of charge-parity or CP-viola- 


h examining the histones of these ihitee 
discoveries since they reveal much that can be 
used to explain CERN's record desing the caity 
1960s. * = 

Ths idea of cadertaking an Sipeninent io de: 


termine whether: the _ electron-neutrino_ and the 


seems. to have occurred more or less simulta- 
neously. to” experitoenters at_ Brookhaven and 
CERN if boot 1950, It was clear thar the higher 
energies of the AGS. and PS would, for the first 
iis to. be carried 


out on & particle accelerator, and the two-neutrino 
experiment was, as one of the physicists involved 


ment-was attempted at-CERN in 1961, but without 
suceess (Weisskopf (71, p. 16]).-As the Director- 


General of the time later observed: ; 


too early - end the first experiment did nat kead to any 
results because of lack of intensity (Weisskopf [73 p. 36). 


andin subsequent years, this failure was attributed 
by the director of one of the CERN Depastments 
to 
a°still inhomogencous and inexperienced group _ faa 
made some mistakes in evaluating the Muxes of particles 
available (Cocconi (34, p. 349)). ‘ 


[Tne PS} had a head start on usa: CERN, and it was clear 


that they would be-at leasi six months ahead of-us,maybe 
even a year ... [T]he question then was: Would CERN bear 
us? And indeed the entre expenmen: was designed at 
CERN ead es siesoet feed) 80 scent sad ee a 


. that-Von Dardel had-discovered.a mistake in the calcu- 
intions and indeed it tured out that the beam as. it was 


the beam was planned for & $-foot straight secuon and-the 
defocusing effect of the magnets mght efter the straight 
section would have essentially demobshed the beam ... The 
other mistake, which is m a sense much deeper and is really 
an indication of the-very seal diflerence in physics philoso- 
phy in [the U-S.] compared to what it was there [ai CERN]. 
was that rather than saying this experiment is s0 important 
let's move it to the 20-foot straight section. the hell with all 


Cancelled _ the. experiment._and of course _we knew they 
would because that was just the style (Schwarts 161, P. Sp 


rapidly been rectified by moving less important 
experiments out of | the way. This would not have 


there were no committess to decide whether you fun of not. 
It_was ust Maurice |Goldhaber, the Laboratory Director] 


and ... his great wisdom prevailed | bid } 
ge ee ee that 


the known asiros vesonnecs = there was a“ “gap” 
corresponding to & new, previously. unobserved, 
pace, The subsequent search for the predicted 


One | of those involved in ihe rival ‘Aanerican ex. 


Deriment recollected his experience of the situation 
as follows: 


paper, we have inevitably heen forced to resort-to @ certain 
amount of high-energy physics.“ jargon” in giving names of 
particles, energy parameters, details of equipment, and so 
on. We have atiempted to keep-this to a-minimum. -The 
interested reader is-refesred-10 articles such a3 thoee-by 
Georgi 37), Jacob and Landshoff [45]. Polkinghorne [56]. 
Salam” {59}, *1 Hooft [42),and Weinberg [70] for recent 
; Popular commentarier on high-energy physics. -— —- 
Theory suggested that the only way to.explain the apparent 
failure of the myon to decay into an electron plus « gamma 
ray was to posit the exisience of two different types of 
neutrinos (Schwartz (61, p. 42)). 


iewards the end of 1962, and agaiti astitned the 
form of a “race” bet user-groups 


i the user-groups of the 
two rival accelerators. At CERN, experimenters 
initially used a beam of pte mesons of momentun 
3.5 Gev/e.t " particle. They 
therefore decided to raise a K~ momentum to 
6 GeV/c, 


but, due to a number of difficulties, this experiment was 


delayed as late as the summer of 1964. Meanwhile the 
Brookhaven group se1_up aS GeV/c K~ beam and early in 
1964 they found a truly remarkable hoograph of a O° 

and decay ... This was a great triumph for the 
new. {SU(3)] theory involving particle symmetries (Butler [8, 
p. 767). 


One of the major diffi culties appears to have 
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involved the British 1S-meire bubble-cha - rer that 
was to be used as the particle detector in this 


experiment. It was reported in 1959 that this would 


be ready for installation at CERN during 1961 (54; 


p. 944). In the event.-its arriva] from: Britain was 
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anus ae es sos ee 
advantage over CERN: 7 

——Itis Clear that one “of the main problems at 
CERN during this early period was the jack of 


experimental equipment. Nowhere was this more 


Gelayed-until 1963, and it then failed to pass its 
initial operational tials ‘on the PS in the early f part 
of 1964 (9, p. 76], The firsi expenmenial run with 
the bubbie-chamber and the high-energy K- beam 
was therefore delayed imitil June 1964. * 

-—The third and. probably most important of the 
early. Brookhaven AGS discoveries was-that of 
CP-violation.’ Two points should-be noted -in 
Connection with the planning and execution of this 


freely admited that no detailed calculations had 
yet-been made (cf. . Fitch (35. pp. 49-51). Whereas 
such a-proposal was acceptable at Brookhaven, 
where ‘decisions on experiments were generally 
made by the Director alone, its chances of negoti- 
ating the formal committee system at CERN would 
probably bave been somewhat slimmer. Insicad of 
trusting the intuition of the two principal physi- 
Sealey 5 Sida Poel noes a ee 


apparent. than in relation. to bubble-chambers 


which, it was beginning to be recognized, had clear 


sult, a great deal of effort was directed in the 
United States 10 the development of this new 
technology. Initially, Europe failed to match this 
Pace of development, as the Director-General of 
CERN admitted in 1963: 


CENN fas Sigaad ballad ike Asmsieas isberaecio aia 
fespect to the use of big bubble-chambers (Weisskopt! [72. 
p. 13). 


=-At ithe time Of the first experiments on the 


CERN PS.in-1960, the largest bubble-chamber at 


the laboratory was a mere 300 cm-(12 inches) in 


size, compared with-the-72-inch instrument in use 
eae a 


ote mies eee ase 


menters at rookhaven were able to start taking 


data-within-~ mere-two months of the proposal 
being written. A second point to note is that a 


sborianad he-time peated 10 monet the A<pat. 


ment. "This detector had previously been xed for 


accelerator at Brookhaven, and. therefore could be 
moved very quickly to the AGS. Indeed, the derec- 
or was installed before the experiment 
proved (cf. Robinson [5S8, p. 620), again 


ing the relatively informal system for planning 7 4d 


executing ¢ experiments then | prevalent at Brookha- 


operational at Brookhaven for the Previous ten 


= (aii poms ncaa es Gus lod dg ce 
emgy K° "beam was that priority was given toa search for 
the omega-minut using ihe anuproton. beam fromthe PS. 
‘This search proved unsuccess{w) because the antiproton 
~ €toss-sections tumed wi tobe lower than predicted. 

7 This was the only hig. energy physics experiment of the 
1960s to be honoured with the award of a Nobel Prize. 


asker: abe ‘was completed. ast 4 French user-group 
brought an 81-cm hydrogen chamber from Saclay. 
However. as.we have seen, the large British. 1.5- 
metre _bubble-chamber did not start nS 
periments until the summer of 1964, three years 
after originally planned. Similarly, the 2 2-metre hy- 
drogen bubbie-chamber- under construction--at 


aaah ill-defined experimental “physics pro. 
fal refered te "ie cigaseee at Ged 


proach which characterized the beginning of re 
search work” (Weisskopf [71, p,.16]), while another 
senior CERN Official later described the ; 


tion for the experimental F Programme as 


tors can be seen as underlying these problems with 
the Physics programme and the Jate provision of 


appropriate experimental equipment at the labora- 
tory. 
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_.First: there is the question of the overall mana- 
gement of the PS ‘consiroction_programme— In 
particular. it could be argued that senior “CERN 
Officials placed 100. much emphasis on early coni- 
pletion of the accelerator (preferably ahead of 
Brookhaven). and. too hitle on developing a suita- 
- experimental Programme (and ihe construction 

fthe So : 


coni that 


most _of the activity [at CERN] had been devoted to the 
construction of the accelerator itself (ibid. p. 349) 


suggests thet he [or one had doubts about whether 
ee balance Depress ceestactien £nd Spares 


Second. it-seems. that CERN's failure to appre- 
ciate fully the-need- for- -appropriate experimental 


and therefore knew full well what was involved in 
mounting “state-of-the art” experiments. In par- 
ticular. their greater experience meant that they 
were. better able to adapt their instruments to new 


tical 
ts to be constructed. lt.is probably not 
without significance -that- what was arguably the 
be carried 


_.Third.. asin any. new. multinational research 
venture, it inevitably took ‘time for an_effective 
organizational structure to develop within CERN. 
particular problems arising in defining the role of 


idea was that, while CERN was responsi 
machine, the beams and detectors would be the 


from each Member State.-arriving at the labora- 


tory with the equipment needed to carry out their 
experiments. Under this approach. CERN would 


be expected to provide only a very limited range of 
experimental equipment for outside user-groups. 
This_may_explain why in the early years mos: of 
the. emphasis_at CERN seemed. 16 bé.on the con- 
beam 


ties-of central -staff and outside t 


a per: 
mitted full exploitation of the new facilities: Very 


Quickly, however, it was found that the “truck 
team” approach “did not work -out successfully” 
(Adams {2, p. 86)). This necessitated the laboratory 
having to develop a_ better integrated research 
policy, with experimental teams being brought 10- 
gether more by their common interest in a particu- 
lar. experiment. and by ti- -xisience of. comple- 


mentary technical skills al by their naGonali ty. 


support. 
took time to adjust to these greater demands placed 
upon it,- attest 
-_ A fourth and 
problems of the PS during i its early years ¢ of opera. 
tion was the commitment of significant sections of 
the European high-energy physics community to 
existing research programmes on national _acceler- 
ators. Three.of the four large. Member States - 
Britain, France_and lialy - were each operating 
(or in the process of building) two national accel- 
erator laboratories, while the fourth - West 
. Conseq 


anc- ee ae _in_the United 
Kingdom, there had been a total lack of co-ordina- 
tion between the domestic experimental efforts ° 


and thote at CERN: at least until the early 19605 


* There were. for example, severe difficulties with separated 
beams in the early 1960s. In addition. there was initiclly a 
tendency lo distinguish work into the categories of “acceler. 
sio¢,” “beam,” and “detector.”-with the result thai these 
* For an evaluation of the scientific outputs fram one of the 
Briush nstional laboratories (DaresbuTy), #2e Martin and 
Irvine [50]. 
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Ne.) Perhaps 
with. _wreater _ Support. fron: -within: its Member 


the fac: ing the — performance of 
the PS will. however. _be left until later when we 
have seen how it fared in subsequent years. 


3. 196 fo 196: The CERN PS comes gt 


i _ CERN and the PS accelerator eniered 1965 ie ina 


years earlier.-In-particular. the- accelerator had 


since 1963 been providing a neutrino | beam: with 


¢ Brookhaven AGS. Equally important;_the 
new 2-metre_bubble-chamber had finally_come 
into operation at the end of 1964. Last. CERN 
had- begun in 1964 to plan a_PS Improvemenis 
Programme, the aim being to increase the intensity 
and_repetition-rate of the. accelerator. and to ex- 
tend the experimental areas. The first effects of 
Phase-1_ of this programme began io be felt-in. the 
te ha ot te foe ear eed under. cx 


fic outputs 
fromthe PS over the- out faa Up to: 1968? -- 


3 ste} oes that -during this period the PS 


and. indeed all other accelerators int the world: 
However, the. total “numbers of. Gilations Tor. the 


cal (about 27.5 percent of the worid total). This 
suggests that ihe two machines had a very si 
overall = ding ih on the sd ad Bs of | 


he- AGS.- producing 46 such: papers. compared 
with 42. It had been particularly successful in 
discovering several new resonances, and in in- 
vestigating. the properties Of Others. more ihor- 
oughly than bad inna achieved. Yet the 


the: AGS.- 


to - vee mistaken.” ‘These results were obtained 
using & missing-mass spectrometer, and appeared 
to show a double-peak structure for the A 3. reio- 
nance. This -was the very first_ indication -of “the 
“split Aj.” as it_came to be known. The finding 
was “confirmed” in several other papers by ihe 
same collaboration, by no less “than four other 
collaborations working on the PS, and even by a 
seam at the Brookhaven AGS. By the time of the 
major high-energy physics conference of 1969 (at 
Cund). it was accepted that 


everyone is now agreed thai the A; is spli 


~ It was oniy-in 1970 that experiments at the 
Stanford Linear Accelerator Center (SLAC): and 
later at Brookhaven and CERN; began to indicate 
tha: the A; was not split after all: By the time of 
the major conference of 1972 (held at Chicago), it 
was reported that the split A, 

Row seems to have been definitely swept aader ihe carpet 

135, p. 336). 


AL this is it 6 simply that the-split Ay was th ihe 
only “1 "of 


ing; there was also the early high value isiaces 


vatron. The Bevatron’s world share-of publications 
and. citations: was--by now declining | rapidly be- 
ca ts ergy relative to the PS and AGS, 


of similar or slightly higher energy at Rutherford 
and Argonne, __ 

__ In terms of “advances” = that is, papers cited 
15 oF more tisties ii One year — the PS had 


result “confirmed” by an experiment at about the 
same time on the Princeton-Pennsylvania Acceler- 
ator. At: Berkeley. the Bevatron gave an erro- 
neously high value for the (K,-K,) mass dil- 
ference, as well as results that apparently violated 
the 2S = 4Q. nile... Finally a 
AGS experiment claimed to 
decay of the » particle. However, the split A; was 


by far the longest-lived mistake. Whereas the others 
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Table 2. ae eee 
Eapenmental high-energy physics. 1965 ives — 
Numbers of - Citations * to Avrage Hoghik cued papers 
experimental work of past eitabons (Sumber cned # nime:} 
papers” - four years per pacer 
published in 
past two wears ae Be et 
: 1966 1968 1966 196% 1966 1968 > 1$ 4 > 30 > 3) > 100 
CERN PS (2k GeV) 200,02 «oS s2ROsia30, 8S Ht 0 
wig Guat (32.52) (25.5%) (2K SF) (2605) ess) (SST) - 
DESY (7 GV) 10 - 40 - D- wo 16 Bi OO. 1. 0 0 
o : ----- (15%) 143%) {O3%) (60%) __ a2 (6.5%) (5.5%) 2 
Rutherford Nimrod(7 GeV) 15 __. 2 - 100 18 60 46 0 0 
ee (2.5%) (2.5) (2.5%) (3.59) = 2 
Other W. Europe 3 - O-- 20 1% 2% 19 0 0 
(3%) (5%) (50%) SR . : 
Toual W. Europe 200, 2S 1620 9.333 0 0 
(44.5%) (RSE) = (36.0%) =: (39.0%) 
Argonne ZGS (12 GeV) 1s $$ $0 258 42.36 6 0 
- (2.0%) (5%) --41.0%) (5.0%) - - - 
Brookhaven AGS(37GeV) 128... 16S... 1270. 1350... 6R 46 1 oO 
AIG SES (19S) ROK) (2854) (100%) 
Brookhaven Cosmotron Q Gev)30 20 200 160 - 36 32 (0) 6 
45%) (25%) = (45%) (20%) _. = 
Berkeley Bevatron(6GeV) 80 98. 780, SOD ROR ] ) 
Bee fetes 125%) (LSE) 17.5%) (10.0%) 
CEA (6 GeV) 15 38 190 - 200 - S4 39 0 0 
(2.5%) (40%) (OF = _ (4.0%) 
SLAC (20 GeV) - 2. - 100.0 45 0 0 
Rees ae (25%) (2.0%) - e 
Other US 35 O- W- 40 37 da 2. 0 0 
- ‘ 45.08) (TOE) _ (6.0%) _ ARO) see 2. 53) LOT) = 
Toial US wo... 480.2710 950 a 9 RS ! 0 
(46.5%) — (53.5%5 (60.0%) (58.0%) (54.5) (83.5%) (ey 
Dubna (10 GeV) oO 3%: 120 £6 it 09 C) 0 0 0 
(6.5) (40%) «= (258) (1.0%) = - 
Moscow ITEP (7 GeV) 1S... BB 0. M17 06 0 0 0 0 
(25%) (30%) = (1.08) (0.5%) > a : : 
Other = rest of world - i 530 - 42 i - 1 0 0 
ae = 0s) == Got --  -- COBH . - 
Total - rest of worid w_.. se es Ce be | A 1 0 o 
(9.0%) (85%) 3.5%) (2.5%) (0.5) (5.5%) 
World total 64S. Bs 4500 smo. 3R 34 150K rn) 
To 00%) 100%) (100%) (1008) (00%) = 00%) (100) 


© Ali citation figures have been rounded to the scares JO. __ 
¢ All percentage figures have been rounded to the nearest 0.5%. 


were corrected within t two to ihree years. the split 
ctl. During this 
time,-- it caused “consternation” to theorisis 


(Jentschke (46, p. 23). ). occupying | a considerable 
i because it cast doubt on the 


otherwise very successful SU(3) classification 
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0 tables T-and 9 of Martin and Irvine (51}3 also appiv. 


scheme for hadrons. The high number of-citations 
eamed by the original split A, paper sugges 
at. the time it Had @. major impact on the scientific 
community. stimulating much experimental _ and 
theoretical work that hee not otherwise have 


Oo. 
ERIC 


Fermilab papers discussed in Paper-i (Martin and 
Irvine (51. p. 201). it is debatable w 
Publication can be regarded as a ve cc 
ticii to the advance of scientific knowledge. __ 

~ To sum v,-, although the PS had a higher publi- 
Cation ut than the AGS over the four-year 


earned a similar total number of citations. How. 
ever, the main difference between the [wo acceler. 
rs was that the AGS _ had -several “major 
advances” to its credit while. the PS had only. one 
(and ‘one which waa later to prove of somewhat 
dubious merit). In terms Of these biblioz di- 
cators. then, the overall scientific of 


Pronounced than hitherto, users of the CERN PS 
continued on ‘average to be less. cxpetiericed in 
sing large accelerators than their Brookhaven 
counterparts, Second, the AGS was by all acc 
the better instrumented of the two accel 
the CERN PS _ was still several years fi 
ing the very large 25-cubic metres } \ 
ble-chamber, and the 10-cubic metres heavy-liquid 
bubble-chamber, recommended by the Amaldi 
Committee (the forerunner of the European Com- 
mittee for Future Ac 

1963. Finally, 


“For a number of years if the 1960s, Brookhaven was the 
finest high-energy physics laboratory in the world.” |26, p. 
247), However. whether this-assessment would have been 
ecoepted in.1968 is less obvious. At the end-of that year, L. 
Van Hove (formesty Head of ihe. Theoretical Physics. De- 
Partment: at CERN and Director-General) under, 
took a review of recent contributions from CERN seccelera- 
tors to high-energy physics. He identified five major contri« 
butior.s, one on the synchso-cycloiron (ihe latesi g-2 experi- 
ment) and four on the PS: (t) leptonic decays of vecior 
mesons: (2) the discovery of several new mesons: (3) the 
split A3; and (4) texts of Cabibbo theory, it-was somewhat 
unfortunate that, of the four contribulions trom the PS. the 
split A, should later be found (0 be “mistaken,” whisc 
several of the new resonances (the S. T and U mesons) 


subsequently came under considerable doubt. 


Storage Rings project and to the eveni more distant 
Proton Synchro- 
tron, Brookhaven staff, by contrast, had no such 


Phase Il began to be reflected ina considerably 
increased accelerator intensity (with a mean inien- 
sity of 1,3. 10" protons per-polse for ihe -year, 
nearly twice the figure for 1965, and ted times 
greater than that for 1961 — “see [10, p..77)). Then, 
in 1971,‘the new heavy-liquid bubble-cham er fi 
nally came into_operai These two 1 dc 
velopments together constituted & considerable up- 
Brading of the PS facility, and for perhaps the-first 


time gave CERN user-groups a significant techni- 


thin. the allocated: budget 
and six months ahead of schedule, as well as the 
go-ahead: (after several years of complex inter- 
governmental negotitations) for construction of the 
Super Proton Synchrotron. All this gave rise to “a 
State of unaccustomed eup' laboratory 
(14, ‘p.31). The first observation 


jon of interactions at 
the ISR was described in the following 


*” Brookhavedi did pui itself forward as 1 
Planned American 200 GeV proion on which. even- 
tually went to the Mid-West (Fermilab). However. the in- 
volvement of Brookha’ 
design-study having been carried 
unsuccessful contender for the site). 
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machine gone so smoothly, Gespite the fact that the ISR is 
= ; the 


To be wiih ihe jubilans phymoists that day when they were 
clocking up particles coming from interactions in the collid- 
ing beams was a rere experience. Not every day does 
excitement break through the surface so openty [14]. 


- To what extent was this mood of optimism 
future for the ISR subsequentiy matched 


gni 
ity? And what effect did the Impr 
gramme have on the scientific contributions made 
by the PS-at the start of the 1970s?-_ 

-” Data on the scientific publications arising from 
the world's main accelerators over _the - period 
1969-72_are given in table ©. As in 1968.-the 


CERN PS continued to produce about 25 percent 
of ‘the world. total of experimental publications. 
This was a considerably greater proportion than 
that achieved by the Brookhaveni AGS. whose 
share dropped from. 19. percent to 14:5 percent. in 
1972, principally because a major conversion pro- 
'2 for the accelerator proved rather more 


more. times in ‘a year) ee 1969 and 1 1972, 


behind in the previous four-year period in terms of 
“major advances'’), and perhaps had even begun 
to edge slightly ahead. 

However, for this period, the most highly cited 
papers came it¢ither from the PS nor the AGS. but 
12 As with the CERN PS_Jmprovement Programme. the aim 

was to increase the intensity of the proton beam. However, 

the problems experienced at Brookhaven withthe conver- 
mon Programme were sO Breat that “she AGS. operated 
initially with beam. intensities somewhat lower than pre- 

shutdown values” [7. p. 47]. 
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é In particular at 

cations Of .@ rising total 
cross-section in hadron collisions were found by a 
joint CERN experiment. The discovery of 


(Subsequently identified with “quarks”). The very 
first results from Fermilab on proton-proton. total] 
1972, and | these 100 were highily cited. ‘And, finally: 
}° portant carly results from the 
ISR: (1) measurements Of . inclusive: reactions, 
which showed the. same “scaling” behaviour as 
had first been seen in the SLAC. deep inelastic 


ratntiey high, publication: output of SLAC over 
this period, it. is } probably. fair_ to. conclude that it 


ited more to the field than those et any oll 
laboratory in the world. CERN was probably ! 

lowed by SLAC; Serpukhov, and Brookhaven, 
the Berkeley and Argonne accelerators sharing 
fifth place. a little ahead of DESY. Thus, just over 
a decade after _the Ps: had come into operation, 


o These papers 

- ment-had actusily t 

'* The discoveries of high irans’ 
fising total cross-sections for pre: ton- proton collisions were 
made in 1972. but. since the results were not published uniil 
1973, they are considered in the discussion of the period 
1973-76 in the next section. 


momentum events | and of 
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Tabe 3 
Expenmental high-energy pi pha 1969-1972 eaianeateetiedes 


Highly cited pagers 
(Number cited n times) 


a>lS na nS 25100 
‘4 


1972 


CERN tSR (28 + 28 GeV) 


13__ 
eee (5.0%) os _ Q0S%) (9.0%) (22.0%) 
CERN PS (28 Gev) 1159 26 23019 0 0 
421.5%) (155%) (5.0%) . 
DESY (7 GeV) 440400 3 0 0 0 
Sesaeee et ssea\ah . : 25%) = é 
Rutherford Nimrod (7 GeV) 32,0«419 C) 0 
Daresbury NINA (4 GeV) 2 2.7 () 0 0 0 
Other W. Europe pe) 2.7 6 2 0 0 
-- -- (S08) (OSB) - - 
Tota) W. Europe 28 28 4t- ee ee 0 
(33.5%) (335%) (22.08) 
Argonne ZGS (12 GeV) 2800 260 7 0 0 0 
eee eee ee (5.5%) 
Brookhaven AGS (33 GeV) 3s 29) 22 i 0 0 
Ss Aves ue hs = - (18.0%) (3.0%) 2 
Berkeley Bevatron (6 GeV) 29 25 8 41 0 0 
; a2). 6 (65%) (5.0%) 
CEA (6 GeV) 300039300 2 0) 0 C) 
—— = -. OSB) = - 
Cornell (12 Gev) 48 4220 4 #0 0 0 
as --  QS%) - ; _ 
SLAC (20 GeV) 61 49 Whe a 2. 0 
ee a __ (170%) 9.0%) (22.08) = 
Other US 25 19 6 4 93. 0 
i -- -- 0%) 19.0%) 358) | 
Total US 0 0206330 (30 COD C) 
(53.0%) : (S70) (47.5%) (55.5%) 
Serpukhov (70 GeV) 270) 48 30 a 
Deere eee as 15.0%) (10.7)* (6.4)* (9.5%) (19.0%) (22.0%) gen 
Other - rest of world 40... 07 0s 0 0 0 
pe nee (RSE) ete : = 
Total - rest of world : 420 14 14 12 eer ree 
(12.5%) (17.0%) (3.08) (8.0%) - (95%) (19.0%) (22.0%) (1008) 
World total 950 1020 5270. $00, D2 173. -21- 9 


Spo A x 
(100%) (1008) (00%) (100%) (100%) 200%) (100%) (100%) 


° The netes 10 tables 7 and 9 
® All publication 


to the nearest 10. 
¢ All perceniage figures have beea rounded to the nearest 0.5%. 
* This is the figure for papers in Western journals only. 


ing the world’s. foremost I igh-criergy physies following quotations from the /972 Annual Report 
laboralory. This elévaiion inio a Position of pre- make apparent: 
eminence was clearly sensed ai CERN, as the The results from European physicisis, working at SERN or 


o.. 
ERIC 


qaais at the Internationsl_Conlerence on -High-Energy 
Phymes which took place at Chicago. Many scientisis ai the 
Conference acknowledged that. in its field, the CERN 


Laboratory 13 tecond to none t hemuschke (47. p. 11)). ; 

For parucle phyucs at CERN. 1972 proved tobe an 
Outstanding year. the pre-eminent place of the Organi- 
zations rescarch being generally recognized. in particular at 


the Bataia.conference. Not only.was the PS physics pro- 
Bramme at_ieast as soteresting asin the past but also there 
were the first exciting results from the ISR [11. p. 26}. 


5. 1973 io 1976: The American revolution 


To what extent were CERN users able to 1 
tain -and build upon the pre-eminent: position 


achicved in 1972? On ‘the: technical side, various 


developments took place t that served to strengthen 


greatly increased intensities of. particle beam 10 be 
obtained. This was to prove_of crucial importance 
for the investigation of weak interactions, in par- 
ticular those involving neutrinos, enabling the full 
potential of- gens ag heavy-liquid | bubble- 


1973. being used-in PS experi 
1975 and (with { 


age Rings (ISR) were oo now coming into their 
own as @ powerful experimental facility. partiy 
because of continual increases achieved” in 
luminosity (and hence in. the number of collisions 
per second), and partly because of the large Split 
Field Magnet detector which began operating in 
1973. 

With the benefit of all these technical mane, 
CERN witnessed in 1973 what _was probably the 

Boas ti 


the > -Gargamelie- bubble-chamber_ on the PS, but 
several other major advances also arose_ from 


experi ts on both the PS and ISR. (The details 


the Director-General was able to report with soivie 
pride that, 


in. 1973 CERN ied major advances in our knowledge of the 


fundamental properties of matter and. of the forces de- 
the behaviour of matter in our Universe (Jentschke 


termining | 
{48. p. 12). 


__ However, the period from 1973 10 1976 was-one 


tole in these developments. 


yas St 
quently filled-by accelerators elsewhere, as can be 


appoii 
bubble chamber (BEBC ~ the Big European Bub- 
ble-Chamber) could only | be used fora few experi- 
ments on the PS in 1973 '* and again in 1975, 
Virtually the whole of 1974 was lost when the 
BEBC magnet had to be completely dismantled to 
rectify a chort-circuiting fault As a result, this 
majot ‘experimental facility - first recommended 
in 1963 as part of the programme to exploit the PS 
= was. in the.event only used for a short time on 
the accelerator (although. as we shall see later, it 
proved invaluable in the SPS _experimental /pro- 
gramme). Besides these improvements in the re- 
ity of the PS, the intersecting Sior- 


‘ably fror the: newly i in- 

ty of over. 5x10"? protons-per pulse. an 
improvement by a factor of 2 over the usual PS iniensity 
Nop 

'* Vanous technical problems were encountered during that 
year: see. for example, |17. pp. 370-71]. 


seen from table 4. True, the CERN PS and ISR 
were still able between :hem_to account for some 
25-30. percent of ‘the worid's publications snd 


and citatior totals for the AGS sagan o with the 
late 1960s (the citation figure, for example, dropped 
from 28 percent of the- world total in 1966-to-10 
f “advances” 

(papers cited 15 :or- more times in-a -year), the 
CERN PS (9 percent) and | ISR (14 percent) were 
some way behind Fermilab (36 percent) aad SLAC 
qs _percent). Moreover, most of the crucial dis- 


coveries which revolutionized the field were made 
elsewhere, particularly on SPEAR, the storage-ring 
operated _by. the. Stanford Linear. Accelerator 
tively cheap secelerator was responsible fot no less 
than 5 out of the 8 papers cited 100 or more times 


in a year that were published during this period. 
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Tabs 2 ioe. 
Expenmental high-energs physics. 1973-1976 * — 
Numben of | Citations © 10 Average Highly cited papers 
experimental work of past Gtauions (Number cited a times) 
n>ISm>30_ n>30 m>100 
CERN ISR (28 + 28 GeV) 40 oO - 3 #0 140 AQS. . be silse 
eicecc secant -GSB)¢ (5.0%) 20%) 9.08) 180%) (7.5%) 12.5%) 
CERN PS (28 GeV) Ww. WM tw 27 ry 2 0 
eel ke | | 20.0%) = (23.5%) (19S) a SH) (6.0%) (7.5%) 
DESV{i)7GeV- 35 » 1x9 0s1M—ié2 2. 1. 6 
_ di) DORIS 144+4GeV) 3.0%) (25%) 125%) 2.8) G.0%) (4.0%) 
Rutherford Nimrod (7GeV) 38 a @ ii 0 5 0 
ee GOB) (SH) (20%) (0.5) = < 
Daresbury NINA (4 GeV) 1s -- 10 c) «0 23001 0 0 0 0 
= (1.08) (10%) -1.0%)— - (0.59) - 
Other W. Europe oOo 50 3130. 340_ 300 Fi Sa 2-- 1 i 
= 7 (55%) (4.00%) (S.0%) (a. SS) (258) (2.0%) (40%) (1258) 
Total W. Earope 4i0 __ “S-— 230 - 2460 350290 SO 
QS.5%) (38.0%) (420%) (32.0%) (275%) (30.0%) (23.0%) (25.08%) 
Argonne ZGS (12 GeV) 100._. MW 4M HHO 22 $- 0 ) ° 
90%) 6.0%) (70%) aS) Ose, - = 
Brookhaven AGS (33 GeV) 155 3 wi wm 28 32 oe ces io 
soit de eee eves (135%) 80%) 11.08) 0.0%) (75%) (75%) (75%) (12.5%) 
Fermilab (400 GeV) 108. 175. MO 10-6 OH 26 9 0 
miler ee (9.5%) (1S.08) (12.5%) S.0F) (38S) (39.0%) (345%) 
CEA GEG - | - 15 . wo 8H” 89 23 2. 22. tL. 8 
_ GiBYPACS(3+3GeV) = 108) 208) (0.58) ; (1.0%) 3.08) 4.08) 
Cornell (12 GeV) 3 we OH a“ A. r) 0 0 
eee (10%) (SK) SB) SS) _ . 3%) _- : d 
SLAC (i)20GeV . _ -. mo - 110 670 930000 420-7100 7 ae 
__(i) SPEAR (4+ 4 GeV) 41.0%) 490%) (10.0%) (17.08) 23.5! (19.0%) (21,08) (31.0%) (62.5%) 
Other US D 0M MM WM 0 Li 2 
pene (45%) (2.0%) 408) F, 
Total US SiS-— 48S 3190 4590 13045 
(43.08) (41.05) (47.0%) (39.59) 
Serpukhov (79 GeV) WO... 12-2 48-49-22 7- t) t) t) 
ee ee (9.8%) (10.8%) (708) 835) (46) FG) 5G) | 
Other - rest of world ws i252 DR i. on) t) 0 
decée * pret, ee (00%) 108%) 48) SH assy : . 
Total - rest of world 25... 5... WM. 6D 86 is 10 0 0 0 
: (19.5%) (21.0%) (11.0%) 9.0%) (5.0%) 
World total 150, 1180. 6780. 7740. 3100330 i97- 67- 26 FY 
(100%) — (200%) = (1ONE)-— 100) 100%) 00%) 100%) (IGOR) 


* The notes to tables? and 9 of Martin and Irvine [Sl] so apply. ==~*~*~*~*~CS 
* All publication ‘figures have been rounded 10 the nearest 5. 

< All citation figures have ‘been rounded to the nearest 10.___ 

£ All percentage figures-hsve been rounded to the nearest 0.5. 

5 This is the figure for DORIS papers only. 

' This is the figute for SPEAR papers aly. 

§ This is the figure for papers in Western journals only. 
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These papers feporied the Gacoverion of she Ifa 


(a discovery shared with the Brookhaven AGS). 
the: psi_ 

charmed mesons D-and D* which were generally 
seen as representing the first_conclusive evidence 
of explicit charm (there had been -easlier hints of 
charm from Brookhaven and CERN). 


that neutrino (and antineutrino)-nucleon cross- 
Sections rise linearly with energy. When combined 
with earlier SLAC work on deep inelastic scatter= 
ing. these” results provided strong experimental 
evidence. thal. “quarks” were more than just an 


-To set againsi this, the CERN accelerators had 
a.number of notable contributions to their credit. 
though. none. seems to have had quite the same 
impact. From the ISR, there was the discovery of 


interesting. theoretical: concept. In other words, 
quar . 440}. 


fies ital experimental evidence 5 and weak -inter- 
actions * put forward by S-- Weinberg and A. 


Salam in 1967. However, perhaps because high-en- 


although _many_ Westen, physicists seem not to 
have been convinced by the result until it was 
subsequently “confirmed” — by —the— ISR 
experiment..’’_ Conrequently, the ISR _ finding 
would almost certainly have had a greater impact 
if. it had been the first of its kind rather than the 
second. Another important, and at the time unique, 
result from the ISR was the discovery of an 


momentum. \-events - (see footnote 14)- which pro- 


vided- Seis S the existence of point- 


is hadronic 
structure had bes Oba daeobes the deep 
inelastic scattering experiments at SLAC) which 
somewhat lessened the impact of the ISR findings. 
Beside these two discoveries, the ISR was responsi- 
ble for several, slightly lower-level advances, in- 


exgy_ physicists had become accustomed to major 
discoveries being made in the United States rather 


Hae 372), some. of: those who: had 
worked in the past in the field of neutrino physics 
being amongst the more sceptical. Nor were the 
CERN management particularly quick in lending 


" Gliding the first indications of a “jet” structure in 


electromagnetic, 2iftec 
theory has been an elusive dream of physicists. since Ein- 


the. ie. particles. produced i in Proton collisions, and Lard 


$. 
In addition, the CERN PS made a acai of 
important advances during this period. One of 


these arose from a 1973 experiment with the 


7 $n 1971. sucha growth of ‘the positive kaon-proton total 
cross-section was reportod from Serpukhov but tc ides of a 
cross-section growing agai gain with energy did not, in-gencral, 
sink in.” (18, p. 67}). For a-turther discussion of the reia- 
Gonship between East European'and Western high-energy 
physics, as well as an evaluation of the scienufic perfor- 
mance of the Serpukhov accelerator, see Irvine and Martin 
[Qi 


stein, and it is the over-riding goal of high-energy physicists 
today This first step, to unify two of the four forces, is 
therefore of vital importanoe, and has been described as “a 
synthesis as profound as shat achieved by Maxwell when he 
uniled the phenomena of eleciricity and magnetism in one 
theoretical | framework”-(Jentechke [48,-p.-23}). The authors 

- of-the theory have since-been rewarded with « Nobel-Prizs. 
'* Ch. '¢§- Barish 15. p. 313]: “It was difficult 16 ‘make an 


is under-cousirained and alternate mechanisms for the ob- 


servations are possible. Rather thao actually ‘seeing’ neutral 
current events, other explanations for apparent events of 
Uus topology must be explained away. For this reason. it 


was vital to book for neutral currenis in a variety of experi- 
ments and reactions with different beckground problems.” 
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ever. as with several of the other major advances 
from CERN discussed earlier, this result was not 
the first Of its type to be published. 
went_tO a bubble-chamber 

Brookhaven AGS acce c er 
mo ing to aru interpre- 
tations of a candidate event, finally published their 
result in April 197S.-Consequently, the Brookhs- 
ven publication seems to have had slighdy raore 


Jeast if judged in terms of nombers of 


85 to whether neutral currents had been established. Those 
who thought they had, had to argue a lot. IT you Jook St the 
early anaouncements of the work in the CERN Cowier. you 
ing the discovery. Neutral currenus were aot believed until 
they were confirmed by Fermilab (Interview. 1981). 
-These early doubts (which were. largely dis- 
pel 974 by the publication of a new and 
more detailed paper on the Gargamelle results) 
may explain why. despite the significance of the 
discovery, the main experimental publication Te 
Porting the résult failed to earn. more 00 


isi 
imilerity between the Brooktia- 
ven AGS and CERN PS: accelerators, there has 


Physics-community as to why the I /psi_ was not 
discovered on the PS (or indeed of the ISR where 
y con- _—sit_ was observed very soon after the discovery). In 
on the disadvantages.) Taken together. the -case of the PS, the reasons were technical as 
_Currents and the linear rise of the toial 
nuclear cross-section ¢an with some 


well-deserved usientific rewards of very’ éelibaraiz_ policies 
and of efforts. pursued over many years. PS these 


policies involved ihe increase of the. 
he : ee 


ws ‘been found (cf. Ting (63): Un- 
doubtedly-it would have been considerably more 
difficult to make the discovery. ai-26.GeV. ibe 
CERN. Hence, in this particular case, the relatively small 
American announcement” on the failure by ihe'Harvard. difference in energy between the iwo accelerators 


covery of neutral curreais warranted ihe award of s Nobel 
Prize. Why it was noe thus rewarded ié aiill a matter of 
speculation. 
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mitied 10 carry out on ioe a eae 
Seperiatont vary memioe to that ComnpaeteD on tS 
AGS: _ 

Unfortunately. “its approval was somewhai delayed ¢: CERN 

as the firs version of the proposal was mot scoepted (bed)? 
AS was. the case with the CP-violation:experi- 
Ment discussed earlier: this missed opportunity 
seems io have been rélated in part to the more 
complex selection procedures then in operation at 
CERN. At Brookhaven, the approach was.com- 


bispectrometes: 
limitations of the CERN committee procedure sys- 


tem and in particular its tendency, in the past at 
least: to shy away from taking risks. _ 

~ At ta why the ISR did not discover the J/psi, it 
should frst b= noted that it did in fact come very 
close.. One experimental collaboration, which 
finished collecting data at the ISR in September 
1974, ‘two months before the discovery was an- 
nounced, obtained results that were subsequently 


found to contain five J /psi events (cf. [19, p-69}). 
One-of those involved in the collaboration, the 


vantage of the ISR in the experiment, to investigate 
the reason for the anomalous result (which was in 
fact due to the J/psi), he and the group were 
side-tracked by the newly -liscovered phenomena 
of high transverse-moment im events and prompt 
leptons (see above). One. of the CERN staff mem- 
bers et ee ie with the ISK summed up 


he didn’t see it. The machine was right, and the experiment 
was right, and even the observation was right. But we still 


= iw poh ha which gv eS he lowing com 
ment by a-former-Director-of Brookhaven Nationa] Labora- 
tory: “At least we cannoi be accused of having disapproved 
‘an experiment here which then gave exciling results 
elsewhere.” (Goidhaber (38, p. £21) 


di’ se he Jp Til Soman imo 


fathre bere (aterview. 1981). 


ae = putea 
the ISR -until later -when its-outputs up to 1982 


have been considered. To-sum up, then. it is clear 


that with a different: approach at CERN - in 


particular, a more flexible decision-procedure on 
experiments. and a more adventurous attitude on 
the part of ISR experimentalists - the course of 
history in experimental high-energy physics be- 
tween 1973 and 1976 might have been rather dif- 
ferent. As.it turned out, this was the period when 
the.CERN research facilites Jost. their. ‘felatively 


centres, and most of the crucial discoveries in 
these dramatic four years were made by machines 


on the other side. the Atlantic. In shori, the 


the discovery of neutral currents, largely an all- 
American affais. 


6 et 1980: CERN consolidates 


ing period: This time, unlike with the PS and toa 
lesser exient the ISR, i full range of sophisticated 
detectors was ready for immediate experimen‘sl 
use.. The four years_also saw imporiant devclop- 
meni ts on the ISR, with the introduction o various 


quae for _many -importar: 
such 28 the study -f “jets” 


range of yeas facilities: and equipmzat 
needed to take_on the task of consolidating. the 
experimental gains made during the previous revo- 
jutionary period. In this section, ea 
effects these various developments had on the 
scientific output from CERN. 
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Tabs 
Expenmental high-energy physics. 1977-1980 * Se —— eee sete asi: 
Citations * to Average Highly cued papers 
work of past citations (Number cited » times} 
four years per paper 
1978 1980 = «1978 1980 «= «1978-198 n>lS n>30 n>S0 3100 
CERN SPS (400 GeV) B_. M 30- sw 127 50 19 - ?- 3 0 
nn cee See (Q2S%)* (BSS) 40%) HSB) Q13€) (43%) Qis€)  _ 
CERN ISR (31+ 31 GeV) 3 sO S80 $4 44 We. 62.2. 0 
es soe? - (5%) (S.$%) (0%) =.S%) (65%) (40%; - 
CERN PS (28 GeV) Ps ee tb, as ,| a i SB 13 222 1-- 0 
oo ee eS (2208) = (11.5%) (14.5%) (12.5%) (8.0%) (4.0%) (7.0%) 
DESY (i) 7-Gev - - - E e - ; 
(i). DORIS. (543 Ge¥) @ «0 -- O-- $7 a8 % 16 4 0 
_ si#) PETRA (19419GeV) (4.0%) (6.5%) 5.5%) SSS) 117%) * (2S) GSS) CESS) - 
RuwefordNimrod(7GeV) 20 i i CialsD Jo 6 0 
= oh 08) = 25%) (05%) (1.5%) (0.5%) 7 
Daresbury NINA (4 GeV) Ww. WW 2» if 07 0 0 0 0 
a (1.0%) 5%) = = on 2 2 = ss ee 
Other W. Europe 3 2 802 sek > ee ') 0 ° 
= - (GOR) 425%) -¢25%) (2.0%) (20%) - 
Toul W. Europe Cn ae | a 7. ae od 37 g3- Ww 8 0 
(39.0%) (38.0%) (34.54) (48.0%) (S0.5%) (56.5%) (37.0%) 
Argonne ZGS (12 GeV) a, a) | ae) ( ae > ae) b 3... 6 0 
: 9%) GOR) (5.0%) 5.08) = (3.0%) (65%) - 
Brookhaven AGS(32?GeV) 0 -— 30... 420 170 27. «16 0 t) 0 0 
svacitee eette dale 485%) G3%) _ G08) GOR) -— _- Zs a 7 
Fermilab (400 GeV) 190 | _.130._. 73 36 Mo. WW. Bsn. Bee 
: (16.5%) 32.0%) (21.5%) (24.5%) (21.0%) (35.5%) (50.0%) 
AWW ye a ee | ce <a 4... 2. 6 0 
__ (ii) CESR (8 + 8 GeV) 35%) 5%) (1.0%) (1.58) (25%) (4.0%) 
SLAC 2 J wie =u Se = : 
4i) 20. GeV 103._.  33_)ssd1280--, OT %- = -6-- fe 1 
Gi) SPEAR 5%) (6.0%) (15.0%) (11.5%) (12.5)' (16.0%) (125%) (7.0%) (5008) 
Other US 15... oe o wn DB 16 0 6 6 6 
(0.0%) = (GS®) (1.0%) (OSB) 
Tota! US $3 34 a: a oe ae 
(47.5%) (43.5%) (43.0%) (100%) 
Se-pukhov (70 GeV) 135 _ 120 «12 0 0 0 0 
Seer a8 - (120%) (40%) = (4.0%) 7 : 
Oiher - resi of wortd M3. 2H 199 Os 1.0 3 - 0 0 9 
wee eee (100%) (130%) _ (258) = : (239%) - 
Total ~ rest of world 230 ©6510 wool 32. (Oo 0 0 
(25%) (27.0%) = (6.08%) (2.0%) 
World oul 111s- 930-8190 35° 30164 aa le 2 - 
3 (100%) = (100%) (300%) (100%) (100%) (100%) (100%) (100%) 
“and 9 of Martin and krvire [St] also apply. 
Hicatio have becn rounded to the neares: 5. 
* All ctation-figures have-bsen rounded to the nearest 10. 
“ All percentage figures have besn rounded to the nearest 0.5%. 
* This is the figure for PETRA papers only. 
* This is the figure for SPEAR pepers only. 
od 
22 


56-399 O0—86—28 
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: Tip pets abe 8 -jonal Ge e a 
CERN accelerators together accounted for be- 
tween 25 and 30 percent of the world ouiput of 


snare umes 8 year: dé DESY colliders (PETRA 
and. DORIS) appear to have done best with 16 
Such papers.compared with 11 for the CERN 


experimental publications in the period, earning a 
similar proportion of the world total of Giations. 
in terms of individual accelerators, the contribu- 
tions. of the ISR and PS declined somewhat from 
the levels achieved previously. but nowhere near as 
dramatically as the Brookhaven AGS (i- earned 


only 3.percent of citations in 1980 compared with 


23 percent ten years earlier) which virtually 


machines and. 10 for Fermilab. At CERN. the 


sridons ome Sable 5, the ISR “was rater cheer 


included the refutation of the “high-y anomaly” 


observed at Fermilab (see Paper I (51): measure- 


dropped « out of contention as a front-line accelera- 
on -the coadiracuce of the new collider: 
ISABELLE. * To some extent, the declines of the 
PS and ISR were compensated for at CERN by 
the Bowing output_and impact of the SPS. wihich, 


tesented a significant improvement ove. those ai 
Fermilab, achieved a particularly high rate of cia- 
tions per paper. (The figure of 12.7 citations per 

paper in 1978 was. similar to that. for SPEAR at 


Stanford in 1978. and for. PETRA at. DESY. in 


1980.) aaut nee oe ail the major accekzators, 


ments of structure-functions and deviations from 
scaling which were found to be in agreement with 
the predictions of quantum chromodynamics (cur- 
rently regarded as the best candidete theory of 


Sent 10 enable the explanation wo be clenel up 
(Until ten, various competing hypotheses had 


an relotively high Ggure | for Se 
It_should be noted, however, that the record of the 
Fermilab machine did decline markedly_in. the 


been propose’, but the date were nit adequate to 
decide between them.) Analysis of these advances 
ee eee 


latter part of the period, not only becsuse of 
competition. from the SPS, but. also because of 
budgetary pressures arising from an ambitious and 
expensive _capital-expenditure | programme (@n an 


other laboratory - 43 ipacenees with 40 at Fenui- 
lab, 36 at DESY, and 26. a1 SLAC. However, ix 
terms of this indicator. the Fermilab machine was 
the most successful individual accelerator. For ad- 
varices Of & Bighy Righer level {papas Gad 30 ot 


ibe ISABELLE proyect has necessarily we the su 
port of the 33 GeV AGS and its experimental programme.” 


=~ As the leader of one-af the main 
Reuuino experiments at the CERN SPS observed, 


I would gladly trade them our. events foe £ their discovery (J. 
Steinberger, quoted in Walgate (68, p. 279). 


The citation records of papers reporting dis- 


2 For example, 8 report on the 1977 international conference 
at Budapest described how “CERN ... was eager to spread 
the word that the "high-y anomaly’ and related ebnonmali- 
ties in high enengy neutrino jateraciions had been wiped oui 
by new [SPS] data” (24. p. 226}. 
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coveries_suggest that these generally have. more 
impixct on the advance of scientific knowledge 
than even the highest-quality confirmatory experi- 
ment. This probsbiy explains why, in terms of 


inxs, it should be noted that mou 


the highly cited p-pers from Fermiiab were 


publi.:.ed-in 1977 (:-ven Gut_of the ten _paper. 
cited more than 3) times. for example), and that 3 
number of them (three oui of the five cited mare 
than 50 times) reported results (Gn <euons) that 
subsequent!;: came under severe Goubt (see Paper I 


denying the SPS one of is best chances of a 


am So 


WES not in a position to mount aii appropriate 
experiment.” In contrast. the upsilon could _al- 


most cetiainty have been discovered on the ISK, 
and, ss. with the J/j-.i three years earlier, ISR 


Se pee aap 
en energy of 9.5 GeV, which is almovi cxactly the 


mass thet the-upsiion was subsequatly found to 


[51)). .He~ce. ca balance, the SPS was probably 
at ajo uve aie Fine aes aa 


collaboration only ‘lounc” ihe signe! again’ a! Ker the Feryas 
lab discovery. So as with the J/pei. the epsilon was in fact 


silon-at Fermilab: barra iop tia A 
electron-scattering at the Stanford Linear Acceler- 
ator Center - the discovery which, in the eyes of 
many physicists, Gnally_1 removed any. lingering 


the unified theory of electro-weak panes as Es 


ALone stage, it did in fact seem as though the SPS 


"ew years eaclier -t Sta 
teralded as providing “seemingly conclusive evi- 
dence for the fifth quark" (Robinson [57, p. 777)). 
Unfortunetely, the “particle” later disappeared 
under a mass of new «lata (cf. (31, ;.. 261}; 7’ thus 


= * jelibeiagh Souerd carton em ined ben dacowered is 1973, this 
Stanford eaperiment waa the firsi to demoosiraic that they 
had just the properties predicied by the Wainberg-Sale-n 
— theory. 


had been observed, say. at Fermilab, they wou)’ in 

Probability have been pu’ -cly announced, just 
as in early 7976 a Fermilab collaboration reported 
on tle basis of just 11 events, the discovery of « 
new particle at-6 GeV (which they named: the 
“upsilon™ = see [22. p. 83]), another particle which 
then “disappeared” in_ subsequent monthi At 
CERN. in contrast, the tradition. has. generally 
been to adopi_a more cautious approach 10 an- 


and the United States, with high-energy physicists 
in the lattez tending to_adopt a more bold. and 
speculative, but at the tame time more risky, ap- 
proach to their experiments, is one to which we 


™ “ai CERN. the energies available from the SPS for the 
counter experiments which have appropviale detector con- 
figurations in the West Area are 100 low for fruitful upsilon 
bunting ~ (25. p. 320). 


@ 


ERIC 


fA Fuitoxt Provided by ERIC 


Fermilah (400 GeV) 
SLAC PEP (IR+ 18 GeV) 
SLAC SPEAR (444 GeV 


SLAC linear accelerator 


Serpukhow (70 Gev) 
Other ~ rest of world 
Total - rest of world 
World total 


(100%) 


& 
(15.5%) 


ai 


(100%) 


31 


3 1320 __ 1060 __ ROO 36 27 0 
(108) (205%) Q0$F) (16.0%) . Ss .- 
0 - : 20 - - 24 a a0 
QS) (054) ze : . 
16: 30 _ 20 300 64 $5 66 Bee. Ae eee 
(407) Af 0F) (5.0%) (6.0%) a 3 (65%) (25.0%) 
| 330 240 130 32 jo 27 0 0 
(2.0%) - (3.5%) (45%) (2.5%) ae = _ B _ 
lo__. 39. 260. 140 | 30 2K 20 0 
(25%) (65%) (5.0%) (25%) 7 a 
17 2620 2160 1710 34 32 3.0 3 - 1 - 
(30.04) (30%) (41.5%) (3354) Qn0t) 25.0%) 
4s. no - 390 - 300 - 12 i4 i3 6 0 
(13%) (5.0%) 654) (6.0%) oe ere a 7 - 
Mo 240 210 | 20 to 09 my 0 0 
(140%) (40%) (40%) (5.0%) 
9 . $0. - SO - $50 re) ti 12 0 a 
(28.5%) 90%) (10.3%) (51.0%) 
389_ on 5190_ 3070_ 30 27 29 oe be oo oat 
(100%) (100%) (008) (100%) (100%) (100%) 


+ The nntes Jo tables 7 and 9 of Martin and Irvine {51} also apply. 


* All citation figures have been rounded to the nearcst 10. __ 
‘ All percentage figures have been rounded to the ncarest 0.5%. 
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will retara in our discussion below on the acon 
affecting the scientific  pertormnance = each of the 


ique-range of ‘Sophisticated detectors 
zed particle beams, has Played a large 
blin 


part i 


American laboratories i in recent years, as we shall 
see in the next section. 


seen belore and is as sure_a_poi Wer a3 any to the impor: 
tance of the extension in CERN's research potentia! which 
the proton-antiproia collider will bring (30. P. 24%}. - 


--Ir event, this major: sacrifice. very quickly 
began: to pay dividends. The first proton-antipro- 
ton ‘collisions were observed in July 1981, bui 
during the remainder of 1981 the luminosity r Te- 
mained low_~ a_peak_of 5% 102? em72s~! was 
achieved. a factor of 200 below the desig igure. 
By the end of the. following. year, 
luminosity had been in 


nic aces this should be sufficient for. ihe 


oS iene ll 
__ kind (33. p. 7]. 


the first signs of the charged W intermediate ver- 
tor boson had been seen. This. was followed in 
May 1983 by the. announcement of the sincovery 
of the neutral Z5 boson, 
__ Besides the commissioning o 
Proton collider. CERN users also benefited from 
several other improvements to the research facili- 


the. WA experittiental area for the SPS. was up- 
graded. while the new Enropean Hybrid Spectrom- 
eter was completed in 1981. Biving SPS users-a 
wide_range ‘of powertul det 


in 1980-and-1981 respectively, eae 2 
The effects of these various additions andi 
provements to the. experimental facilities t CERN 
are readily apparent in the figures in i 
take first the output of experimental pub! Ons, 
although the number yielded by the SPS dropped 
slightly. in -1982. (reflecting the shut-down a year 
eatlier), this was more than 
€me rgence of the first results from the-proton—an- 
tiproton. (Pp) colli and -by-an-; appreciable in- 


IIL [Sz})- Consequently, whereas papers from. the 
Ens accelerators accounted for 24.5 ape of 


8. M 
1982, finally catching up 


in the meantir tthe SPS had 


nt increase in the overall impact of ISR 
Papers from: 75 percen of the. world ‘otal of 
citations in 1980 to 10.5 percent in 1982. 

> The figures on ciunions Pet paper suggest that, 
in 1982 and 1982, CESR at Cornell and PETRA at 
DESY. were the machines with the highest average 


4 ag fig- 
6 in 1981 (although i it 
in 1982), comparatively 


a r that had only been operat- 
ing for five years.-This somewhat rapid ab- 


solescence can perhaps best be explained by ihe 
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fact. that, - 1980, the Fermilab: “accelerator - = __The. last two columns or table 68 Bive the num- 
which had been operational since 1972 - and the ers publis! 
SPS_had between them carried oul mos 
i portant. wo! in the 400 GeV. energy-range. us 


six- such papers, - followed: by. the id with 
proton-antiproton collider with three, the SPS 


and-CESR with two each, and the ISR with one 


rly (Fermilab had no such papers). If the figures “_: 
illuminating in this respect since they show an CERN -anc DESY are combined; one sees that 


increase from 3,8-in 1981. to: 4.7 in 1982. The latter Western Europe produced 80 percent of the papers 


is remarkably high for a machine that has been cited 15° or more times, while the United States 
operating for twelve years (far higher. for-example, managed only 20 percent. A similar picture emerges 
shan the very successful Brookhaven AGS in its in the case of “ major advances” (papers cited 30 
twelfth year - sc¢ table 3), and must ‘surely be 
linked to the fact thatthe ISR-was until its closure i 
in 1983 always a unique facility. Given that no had just one (from SPEAR), this deing t the first 


larger proton-proton collider is likely to be com- time ever that a CERN | machine thas Jed ed the world 


CERN and DESY give good grounds for sade 
that -the onset of the 1980s had witnessed a 
European renaissance in experimental high-energy 
physics, even before the discoveries of the W and 


Z particles were announced in 1983: 


ponents a frontier research ae in high-en- 


ergy physics. 
Table 7 
Comparison of the scientific ouspuité of the TERN PS and Brookhaven, AGS accelerators: 161-1980 
1961-3964 1965-1968 1969-1972 1973-1976 
———— PS AGS PS. _AGS. PS. A535 ___PS. 
papers * 21.5% 80 = 28.5% 19.5% 25.0 165% 22.0% 105% 175% 55% 
Citations to - pee Seen Sate Baden sey wydes WSEer, seer ates ee 
recent > work * _ 14.5% 23.08 27.5% 2758 22.08 205% 170% 10.5% 125% 4.0% 


19 6.0 3 $s 25 32 25 28 23 22 
Number of papers =n 315 -10.08 2708 «=6305% = - 28.08 15.5% 1808) = 9.H 75K BOK (OOS 
Cited » iimmes® - _. (@3B)S (VOB) (ATES) (9H-OB) (UF-LOBW) (175%) (13.0%) (75%) (7.08) (0.0%) 
No. of times achieved n>30 60% 295% $58 66.5% 50% 50% 60% 75% 408 008. 

(70%) (55.0%) (13.0%) (605%) (4.5% (2.5%) (7.0%) (11.5%) (4.08) (0.08) 
n>50 0.08 25.0% O5% 109%) 00% 00% 75% 758 70% 0.0% 
__.... (0) — (57.0%) (0.0%) (100%) 0.0%) (0.0%) 12.0%) (11.08) (43%) (0.08) 
n>100 0.0% 100%) - - O8% ONG 00% 125% 00% 0.08. 


(0.0%) (100%) (0.0%) £6.08) (0.0%) (19.0%) (0.0%) (0.0%) 
* All Ggures are expressed.as a % of the relevant world total, and each has been r’-nded to the nearest 0.5%. 
° Work published during the preceding 4 years. 
‘ ‘The figures in brackets correspond to those for tie number of times papers earned: * citations in one year. 
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iis sobrall gutecpancs ol tac CERN accede: 
tors 


8.1. The Proton Symchvotron 


= -_ Having analyzed in some detail the historical 
development of experimental high-energy physics 
over the period from 1961 to the end-of 1982,-we 


are now in a posilion to draw some more general 


ic perfor- 
mance of each of the principal CERN accelerators. 


Table 7 summarizes the data on the outputs from 


the first of these accelerators, the Proton Syncaro- 
tron. relative to those from the rival AGS machine 


appreciably more experimental papers thas the 
AGS. However, this is the ‘only respect in which it 
consistently outperformed the AGS, since the 
average impact of those papers was in general 


100-times or more in year) and “ “ major advances” 
(those cited 30 or 30 times or more), particularly 
during | the 19€0s. Certainly, is in the case of “ads 
overtook the AGS in the. mid-1970s, but it had 
been well behind in the early years, Hence, when 
taken together, these indicators Strongly suggest 


cessful than the PS. over the twenty-year period 
under consideration. As noted at the time of its 
twentieth anniversary, the AGS 


bes bees eo te met prodive of scan 


physics, counting amongst its discoveries the muon. neu- 
trino, charge-parity violation, the omega minus, the J/pei, 


= ——— Peortaaki o : 
(1) Discoveries Deaeka et 


Brookhaven AGS 9.50.1) 
anes ery 
CERN PS 7.3(£0.2) 
(n= 6) 
CERN isk 6. 4 0.3) 
Wa %). 
CERN SPS $.% +03) 
(4 = 68) 
Fermilab 400 Gev 7M +0.3) 


:_420.1) 9M£0.1) 
(a ~121) ("= 169) 
6.0.2) 6% 40.1) 
(a= 83) (x =169) 
5.% 202) 6.1, + 0.2) 
(7 ©133) (n= 169) 
3.60.2) $.% 40.1) 
(4 = 101) (9= 169) 
7.1(40.1) 7.20.3) 
(x 123) (n= 169) 
2.0.1) 2% 0.1) 
(no 149) (4 = 169) 
7.2202) 7.2% +0.1) 
(92120) (n © 162). 
§.( 0.1) 8.5( 40.1) 
(4 = 8h) (4 167) 
6% +02) 7% 0.3) 
(42132). (4 =167), 
8.2% 0:2) 8.2% +0.) 
(2= 100) (4 = 167) 
$.% 0.2) 6.1( 40.1) 
(a 121) (49167) 
3.5 +02) 24: +02) 
(4 © 147) («= 167) 


SE 
bottom. . The Tankings are based'on the relative ouipuis Irom the scoelersiors over-their entire careers up to the 
with the high-energy physicists in late 1981 /early 1982. The figures in brackets indicate the root-mean-square 


vanations, giving some approsimate ides cf the divergence of opinion within each group. It should be noted that these rankings 


were normally made by interviewees in written form. 
* The sample size of each group is denoted by x. 
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(a ceased arse GN beam ie pa 
tesonances 132, p. 242]. 


_- Indeed, the AGS can b 


i Wile Stes ence 
ipa by the indicators listed in table 7 is also 
well supported by the results obtained in inter- 
views with 182 high-energy physicists (see Paper 1 
(51) for details). Interviewees were invited to.assess 
the overall scientific fic performance of the world's 

‘incipal p » accelerators on a 10-point 


scale (10 = top, 1 = botto 


Coveries; and | (2) experiments involving more pre- 
cise measurements of known particles and their 


Table 9 


properties but without. discovering anyihing new. 
The results_are given in table 8. On this 10-point 
scale of achievement, the AGS was judged consid- 


erably ahead of the PS in the case of the. first 

criténon.(92 compared with 6.9), a-view, it should 

be Stressed. held by users of both scedleraiors.— 
i however, 


5 considered - ihe PS wer euied BS cc 


at-scale, somewhat ahead of the AGS on 


this result - the total number of publications, and 
the average citations per paper. As can be seen 
from table 7 above, the AGS. had ; rather-betier 
record ini terms of the latter. 
vantage appears to have been outweighed by the 
far greater publication output of the PS, particu- 


Faces epning the relative ae perforant he CERN PS and Bosthaven AGS (8 inieviwes bong i ai 


to have been important *) - 


Of AGS(%) of PS) AGS and PS (8) interviewees (%) 
of 


(m= 125) 


More bold, speculative ethos-of US physicists- $5 


a 
2 2 
3. sd better experimentalisis in 1960s. 2» 
4 Higher work ethic and.more competitive 2 
= attitude of US phyzicists _ 2 
5. Europe's “missing peneration™ « of 19 

physicists - wartime emigration to 

US of many of the best 


sow commie, over-coneervaie chat 


“ 6s 31 
a a % 
2 29 3% 
20 33 20 
ia » ii 
13 Py 7 
14 ua 16 
14 12 10 
“% 59 31 

4 6 10 


ooo eee 
> ‘These figures represent minimum values oaly since they arc based.on » content analyiis of afitwers 10 a general question concerning 


the factors structuring the relative scientific performance of the two accelerators. The sample size of each group of interviewees is 


denoted by n- 
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larly in the 1970s. when the “ ‘chations per paper” 
figure was not-very-different for the-two accelera- 
tors. As-a result, whereas between 1961 and 1964 
AGS: experiments -appear to have had a Signifi- 
cantly greater impact (at least in terms of the total 
numbers of cit feed ean eaclog mae avery 
similar record in the period from 1965 to 1968. 
and then the PS edged ahead. The gap. widened 
considerably diiring the 1970s as the output from 
the AGS declined. citations to: work published 
Over the ‘preceding four years falling from 20 per- 
cat of the world total in 1969-72 to.4 percent in 
1977-80. Thus, over their. entire lifetimes, the re- 


cord. of the PS was probably slightly superior in 


antic performance of the PS and AGS high-ea- 
ergy physicists were also asked to point to reasuiis 
for those differences. Even. though this question 
was unstructured, in the sense that interviewees 
were not. prompted as to all possible reasons for 
the. divergence. in performance,. the responses 


tended to conform to. ceriain patierns, and it was 
sasiber af separate cabaeies Ghnouph ie fae 
though. 


communities of the AGS and PS. paces. 
Portant here was what many interviewees (in both 
Europe and_North America) identified as a more 
bold, speculative ethos of physicists in the United 
States, where the research system was regarded as 
formally allowing more scope f.¢ individual initia- 
tive,In Contrast, physicists in Europe were seen as 
adopting a more solid, sale, aud conservative ap- 


mately half those questioned on the issue, saps 


tather more (65 percent) of those who had direct 


experience of using both the AGS and PS accelera- 
tors, 


Another. extremely i important factor appears to 
have been the generally higher level of experime:- 


particularly i in the 1960s. Many Brookhaven users 
had previously carried out experiments on the 


238 


laboratory's 3 GeV Cosmatron.or the 6 GeV 

Berkeley. Bevatron, while the experience of most 

European experimentalists was limited to accelera- 

tors of 1: (GeV energy or less. ‘The jump involved in 
joving the 


33 fava AGS. Consequently, users of the PS inevi- 


exploit the socelerator than their counterparts on 
the -AGS. This factor was again particularly 
stressed by those who had used both accelerators 
(47 percent compared with 36 percent of all inter. 
viewers). Sce 

_. Other interviewses (30 percent) were rather tess 


ing 
through both the Second World War and emigra- 


tion to the United States in the 1930s and 1940s 
(11 percent). | 

Besides «ese five. user-related explanations, a 
sixth factor identified was the “luck” of Brookha- 
ven. in choosing the right experiments (17 percent). 
Some of those specifying this factor did, hc 


presen percent). 
factor 8 in table 9 (CERN's tendency 1 to “over- 


engineer” - for example, constructing highly 


3 sm sno te euscon shove ta te onion. op 1961-44 


universities have a greater cole in providing equi 
consequence of this, it was argued, is that / 

searchers. have generally been more ix-control of their 
experiments, and, because they understood the equipment 
better, were able to undertake rapid modifications when ihe 
need arcee. 
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sop aicated deinciors ‘het. came into operation 


> Tater than: less advanced: instruments 


some committer structure. than at the US ‘fustional 


been slower, the responie 16 5 Dew initiatives some- 


oe 
06 ¢ at of over 50 


believe that the quality: of the Brookhaven ‘mana- 


Bement was in no small way résponsibletor ihe 
Position of advantage achieved by the AGS over 
the PS. However, these two factors also need to be 


saicrng lor & uige ausske ct Gonnssboaln a 
persed users, CERN inevitably was under great 


Pressure to provide a service that was, above all, 


{4. p. 180}, emphasis added). 


AS a previous Director-General has stated, CERN 
has tied to 


faast es MB igmaes atte cat ae be ting ofa wie 


—— interests has resulted; it was argued; in a 


acca For. —— in Europe, for vari- 
ous. s. historical - reasons, an. academic. culture has 


risky approach, adopted by many young. Ameri- 
cans (see factor 1 in table 9); an approach which 
may fail, but sometimes succeceds dramatically. 
One senior experimentalist, who has worked on 


Benerally greater reliance on a sometimes camber. 


# The author probably hed. in. tind here the experiences of 
Brookhaven | inthe late 3 1970s with the ISABELLE projeci. 


required could be manufactured. ‘Upeeiccun be bambi 
did not pay off, with the resull that there were considerable 


delays and escalating costs -- see, for example, Broad (6}. 


= Ut [52]. The project was finally terminated in 

i winks claivi kadai wowace likely 
technical problems when it embarked on the ISR-project is 
Open to dispure. As the time the JSR began operation, its 
construction was described as “doing something entirely 
Rew whose successful operation could not be guaranteed” 
(114, p. 31], emphasis added). Similar reservations might 
alto apply lo the more recent_proton-antiproion collider 
Project, where it was by no means certain that the design 
tuminosity could be reached. 


in a particular social Siluation. To get ® tenured job-in-a 


career-optimization. ti leads io researchers in Europe work- 
ing on (Precise measurements'-lype experiments, rather than 


igamma, of work (tnterview, 19815. 


To sum up, most_of the factors believed by 
high-energy physicists to account for the dil- 
ferences between the scientific performance of the 
PS and _the AGS extend far beyond the perimeters 
of the CERN laboratory itself, rel; i 
user-community or to the wider 
text of the centre and of research in Western 
Europe. Only factors 7 and 8 relate directly to 


Oi 
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pact. The same is true of the ISR, w 


certain broader features of Encpess society dur- 
ing this Period. 


82. _The jam Storage Rings and the Super 
Proton Synchrotron 


Aieasue petted tee ca 


1972: - --4 percent of the world total-in that year 

Scapa CURA. punceni ier tha 1S: Ta Goel 
“advances” and “major advances.” the records of 
the two acceleraiors were virtually identical. 
Serpukbov_ was, however, responsible for the sole 
“crucial discovery” (cited more than 100 times in a 
year) of this period - the discovery of rising 


is nowhere neat “as straightforward a task as for 
the PS, where a direct comparison could be drawn 
with the AGS because of the similarity in energy 


oe 
the table is ‘one of clear Fermilab dominance, 


four years after the American machine. (This was, 
as we have noted earlier, a rather large Icad to 
concede to Fermilab, given that much of the 
physics in a new energy region tends to be carried 
out within the first few years of becoming accessi- 
ble.) Nevertheless, ‘there are. grounds for. believing 


(gaining 100 or more citations ina year) where the 
ISR did better (asa result of the publication 
reporting the. discovery of rising total cross-sec- 


that. valid. comparisons _| between these and. other 


ble to examine the: scientific performance of 
Fermilab before and after the SPS started Operat- 
ing to determine what effect this new accelerator 
nae ae 


fic community. For. the first four years, 


ore publications ‘gained. a relatively high 
‘CPP i 


figure v was ° 7. 7, having been a as high as! 10.7 i in 1970 
for the first two years of the accelerator’, opera- 


Giana @ Geen ee 
tions and highly cited papers of the four major 
Proton machines. However. some of the mistaken 
results from earlier experiments (see. Paper I [51)) 
were beginning to be recognized, and these should 
be taken into account when comparing its scien- 
fific_ performance with thai of the SPS. As we 
noted earlier, it is-significant that the SPS. figure 
for citations per paper {fell very rapidly from 12.7 


Sp Spa operat tealane Thus; 
while the SPS: was undoubtedly responsible for: a 
number of major advances (cited 30 or 50 times ii 
a year) stemming from sophisticated second-gener- 


IC 
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in.its-early years. In particular. it should be noted 


that. while the ISR ; was responsible for one crucial 


the period reviewed. 


-~-In summary. then, if these four accelerators aré 
compared in terms of “discoveries,” the Fermilab 
machine must undoubtedly be. placed firsi,. fol- 
lowed by the ISR, and. with little to. choose be- 
tween the aggregate records of Serpukhov and the 
SPS. The fact that. the SPS achieved in the. four 
Yyeass up to 1980 approximaiely the same number 
of papers cited more than 15, 30-and.$0 times in a 


years (Serpukhov has contributed very little since 
1972), would probably tip the balance is favour of 
the SPS being -ranked: third, slightly ahead of 
v1 Tis gp back to the peer-evaluation 

earlier in table 8, one sees that these 

rankings are-in close agreement with those made 
by high-energy physicists: Fermilab was. given a 


score-of 7.2 on the 10-point acale for discoveries 
(though, it should be noted: some way behind the 


Table 1 : : 
aClO aa the relative scienufie pettorimance of the CERN ISR (% of interviewees believing this fecior to have beea 


a -2eeee 177 meee eee 
Users Others () 


of ISR{%) (484) eatin (%) 


Brookhaven AGS), while the ISR. was scored 611, 
the SPS 5.7, and Sc rpukhav2.7. > = 

As for the relalivé positions of the four mac- 
hines in terms Of .“prease. measurements.” the 


results are rather more difficult to interpret, largely 
because of the : differing periods of time over which 


SPS has yielded more publications than the three 


other machines, and- -that from 1978-SPS publica- 


a seore of 82. while the ISK war tes raed see 
highly at 7.0 in line with its consisiently high 


» These results if. all Uikelihvod exaggerate the -difference 
between’ the contributions made by the Soviet accelerator 
and-the other three. Part of the difference may 0¢ atinbuta- 
ble to ae ignorance on the part of. many Western scienusts 
of a | the reaults from the Soviet machine ~ see Irvine and 


Manin (43). 


ae  Solidoangie Sraewenl 


5. Rmoasch ogress cs feameasead oe cial 
Mrateay (¢§ (oo many small experiments initially) 


6. Early | Eenty hinineity 100 low for successful 

7 Probe wb aia Geir pit Fi 
ry 

9: turned out not to be « very exciting 


___ énergy region 


nR “o 55 
“ 19 2B 
28 19 21 

3 2B: 1 
J 13 ” 
u i7 21 

7 TT 10 
“a “ a2 
7 5} 27 


* Minimum estimates only (set note @ to table 9). The sample size of each group of interviewees is denoted by a. 
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number of citations per paper. Fermilab. despite 


down by many interviewees because of several 
earlier mistaken results, scoring 6.1, with Serpuk- 
hov again tanked last on | 


CERN ISR and SPS, we can now move: on to 


ae factors that have determined their 
fi 


In particular, its luminosity hss been improved by 
several orders of magnitude over “ils lifetime. far 
surpassing the original design specifications. There 
is thus some justification to ihe Claim that 


‘The :5R is widely regarded as the most perfect example io 

date of the accelerator builder's art (27. p. 231). 

However, there -is, -as has been -seen, a 
widespread: feeling among high-energy physicists 
that, as a research facility. the ISR was for some 
years not exploited as successfully as it might have 
been, Somehow missing the Opportunity. of mabing 


silon. This sense of disappointment that the p poten- 
tial of the machine was not fully realized (which is 


tlie performance. Factors 1 10-5 relate-to: prob- 
lems aes with the ; management of tl this facil- 


critic:sm, especially from ISR users (with 72: “per: 
cent citing this as a major problem). One of the 
CERN senior staff closely connected with the ISR 
recalled the évents surrounding ihis decision in the 
following terms: - 


The Split Field Magnet” is the main detecior on the. ISK: 
The design of the device as originally instrumented em: 


Phaswed forward and backward angles, not large angles. So 
it _mussed the J/psi exactly. Ai the time, CERN manage- 
mt Was Convinced that _no particles would be found al 


ossible spectrometers, bu they chose ihe wrong one. Ai 
One stage. they considered doing two detectors, one for 
looking at large angles. But they decided to phase these, and 
leave the large-angle one Ul later_They thought the ISR 
had the € srgy region 10 itself. They forgot abou the 


Competitive aspect, and Ting [the co-discoveres of the J/psi) 
at Brookhaven. (Interview. 1) _ 7 


large-angle detector was: also probably related to 
factor 2- in -the table, the judgement by some 
high-energy physicists (23 percent of these inter- 
viewed) that the ISR and its experimental pro- 
gramme were not given sufficient priority by 
CERN management. The most Gilical of those 
Physicists believed that the ISR was regarded by 
some senior CERN officials as little more than. a 


(see Paper 1 {51}). In-1973, the Director-General 
reported that Preparations for the SPS were 


beginning to-dominate ur planning at CI CERN ‘since it 


coauderable manpower and financial demands on Labors- 
tory} . . Genuchke [48, p. 23). 
While most Of this heavy burden fell on the PS 
nentz! programme, - ISR did not A eacape 
entirely. — 7 - 
qi percent) w 


there being no overall strategy for exploit- 
ing the accelerator and addressing the outstanding 
theoretical questions of the day (factor 5): As one 
Physicist commented. 
They t use the ISR. in the samme way as she PS. with 
hous of small experiments. (Interview, 1981) _ 


tion, the early experiments were rather too o crowded 
together = there were at one stage four experi- 

ments at just one of the intersection regions. Such 
problems were perhaps compounded by ihe divi- 
sion of opinion that existed (and still exists) within 


Pal 
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Table 12 Z : == 2 
Factors explaining th relative scientific performance of Fermilab and the CERN SPS (% of interviewees believing this (actor 10 have 
been important *) 
Usernof - Users - Users of. al -- 
Fermilab(%) of SPS(%) Fermilab and SPS(%) interviewers (€) 
_ (a2) (n= 56) (ae7}" (a 21%) 
1. Fermilab had 4-year lead - SPS too late % ET 2 38 
2. = % = ee 
“» % 43 39 
3. _ = : 
10 9 14 ll 
4. _ a 7 
i “a 35 43 so 
iS: a 2s = 
15 32 2 6 
6. . _ 
x) 32 2 3 
7. 2 
‘s 15 13 2» 4 
a 2 2 3 
ground with experiments, given the resources 21 14 14 13 
9. Energy range <! both accelerators turned out 10 oe se 
be relatively unexciting 13 5 10 


© Minimais estimsie oaly (sec owe a to ble 9). The sample size of cach group of interviewses is denoted by a. 
* This is a very small sample site, ars the % figures may vot therefore be statistically very significant. 


the laboratory over the value of the ISR as an 

experitiental facility. A-not_insignificant fraction 

Of physicists believed the ISR to. have been the 

wrong choice of machine (factor 4). arguing that 

ae protons is not-as fruitful a research tech- 
target” 


cenwal in limiting the accelerator's scientific per- 
formance, particularly in. the early. years: 21 per- 
cent.of interviewees pointed out that the lumina: 


aah involve: the i inter- 


aterpret ~ 
sation of sues cverks-with divws ciher quarks 


| percent) put forward 
the. view that the ISR had been the victim of bad 
luck, in the sense_that the energy range which it 


ityof the ISR’ was initially too low for certain 
types_of experiment (factor 6);.and 10 percent 
specified various 10 technic anical _dilficuldes ee the 


argued tha: because | the ISR was the world's first 
proton collider, physicists had to start from scratch 


2 ip dees acark ad Se dade ucsienpenioes sg. 


SPEAR, wers ableto draw upon the experience gained from 
earlier, . sealer Fings at | Frascati, Orsay, and Cambridge. 


capability. and some of those interviewed pointed out that 


this was pot available until several years after the ISR began 
operating. 
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Teconcile with the fact that the > J/psi_and- the 
upsilon - the two most highly cited discoveries of 
the 1970s, and arguably the two with the greatect 
- were there to be unearthed by the ISR if 
appropriate experiments had been carried out s00f 
enough. _ 

Let us now move ‘on to“carisider ihe. factors 
underlying the scientific ‘performance of the SPS, 
especially in relation io its Fermilab rival. AS we 


Proved somewhat more successful in terms of jor 
advances and discoveries, while the SPS 


better record (according to the peer-evaluation 


eplais Fermilab's beter record: trom major ad- 
vances. Well over half of those interviewed (58 
percent) felt that the Fermilab lead of some four 
and a half years proved too. big a handicap to the 
SPS (factor. 1) To many, the subsequent difficul- 
ties for the SPS could have been 
early as 1973. it had been rene out 
lead . represents - 


nes Ry sinteresting® lifetime of nasi high enerty 
ph ~ which raises the Question of whether 
there will be significant. diphesigemai for it to pursue. 
(Hammond 140, p. £120)) _ 

| the: recognition iy. CERN in 
1976 of the significance of Fermilab’s lead that led 
them to consider, even at this early stage, other 
possibilities for exploiting the. SPS. It was as a 
response_to this that the proton-antiproton col- 
lider proposal was put forward and approved rela- 
tively quickly (cf. Van Hove [66, p. 31). _ 
_..In addition, this extensive lead plicatio: 
for. the type of ex 


turn, 1, “partly- a function- of, the different styles: of 
American and European researchers (a factor dis- 
cussed earlier in connection with the AGS and PS 
accelerators), and partly the result of deliberate 
Policy choices. mace by wie two. laboratories. At 
Fermilab, they decided to take full advantage of 
the lead over the SPS by carrying out a broad, but 


“cream off” any major discoveries that could be 


made. in. the ‘energy range. The discovery of di- 


experiments. Though this often meant. repeating 
Previous Fermilab work, these experiments. have 
tended to provide definitive séts of statistics, as 
well a clearing up. the mistakes made by the 
American accelerator. Factor 3 - the problems 


Ge ee ei 
resources. and technical support available _at 
CERN, this factor (4) being mentioned by about 
Me ee aes ot oe 


ning its experimental programme, and was forced 
to shut down the accelerator for extended periads.) 
__Also_of great importance ‘has ‘been the high 
technical quality of the CERN SPS: with the 
Fermilab machine so far ahead, there was little 


* ta addi | 4A percent of ihe physicists completing the 
attitude ‘Murvey (described ip the final section of this paper) 
agreed “with the statement, “CERN provides European 
high-energy physicists with a better level of technical sup- 
portend. [acilities than that enjoyed by Americans at their 


National laboratoties”. A mere 2 percent disagreed. 


2” ‘See table 5 in Paper 1 [5t}. 
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@ramme of_the facility. The SPS was therefore 


Fertiilab) ‘alio felt that: in its effort to “skim the 
cream,” Fermilab had tried io do too many experi- 


built very carefully, and is_as a result..a signifi- 


ests iba GORY. By the end ot Ube Hest Toor 


Canily. better accelerator. Thirty-two percent of 
SPS users and 15 percent of Fermilab users identi- 
factor 


moisture from the humid air with the result that 
over half had to be replaced, some. more than 
once. This generally. lower reliabiliry at the 
American centre can be understood parily as a 


result of long-term financial pressures on_ the 


jears of operation, 152 experiments had been com- 


experiments with greawr resolving power (Mew {53. pp. 
196-97). : _ 
__ Finally, we should ante: that 10 percent of in intes- 
viewers Bi nena a feeling that, . in _ retrospect, 


laboratory, ** and. partly as.a consequence of the 
philosophy of R-R. Wilson, the lirst Director of 
Fermilab - a philosophy which involved _ 


funds should: be invested in eccalerain _develop- 
ment to ensure that, once the SPS came into 
operation, the Fermilab facility could be rapidly 
upgraded to give it a renewed advantage. over 
CERN. Some of those interviewed (about 14 per- 
cent ~ ste factor 7) clearly felt that 100 high a 

i ion of Fermilab's boii budget had 


ber of interviewees (13 percent, and 21 percent at 


” * CLs. pay}: 


h 33 percent) s 


Possible comers (0 save time and money has not paid off in 
the long run.” 

© Ci. Metz. [53. p. 196}. The Doubler /Saver is a major project 
(0 increase the energy of the Fermilab accelerator [rom 400 
GeV 10 berween 500 and 1000 GeV. Paper il] [52] gives 
further details. 


ing their larges: investment of the 1970s in exen- 
tially identical accelerators. 

: ‘This concludes our analysis of the factors de- 
termining the relative scientific performance of the 
three_main CERN accelerators. The only remain- 
ing task of this paper is to synthesize the principal 


points arising from our assessmeni of CERN over 
the past twenty years, in particular drawing out 
any Jessons of televance to the subject of Paper Hl 
[52], the future prospects for CERN. 


9. CERN’s past performance - an overall assess- 
ment 

__ Before summarizing the main_conclusions that 
can_be drawn from our assessment of CERN's 
past. scientific performance, it perhaps ‘needs re- 
emphasizing that certain aspects of CERN’S activi- 
ties have nor been evaluated here. We have not, for 
example, Commented upon the very substantial 


final section), 72 percen of those 


has Been @ iendéncy in the pai io accepi 100 may experi- 
ments, with the result tha: experiments have oft. been 
rushed of prematurely cut shor”; only 8 percent disagreed. 
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technical conitributions to high-energy. physics for 
CERN HORE to: 


: ion-of proportional wire chambers. the 
development of streamer chambers (following up 
the pioneering work of G. Chikovani at Tbilisi in 
the Soviet" Union), liquid argon detectors. various 


of stochastic cooling. Nor have we eval ated the 
extensive coatributions made. by-CERN theorists 


=.40 the quark. model,.to Regge-and-Cabibbo 
chromodynamics, 


theory, to quantum 


in passing 
the participation of CERN: physicists in experi- 
ments at Serpukhov, and in particular their'role ifi 
helping make the crucial discovery there of rising 
total cross-sections. : 

~- Furthermore, it should be noted that CERN 
has been responsible for severa! wider coniribu- 
tions besides helping io further our knowledge of 
high-energy physics. These have been described in 


It bas been * "a source of Europes spirit’. fi bas played a hey 
tole inn blishing ‘the stature of European science. [i 


~To tke the frst of these. there was, pe deat in 
the minds of the high-energy physicists we inter- 
viewed that CERN has substantially stimulated 
international co-operation in science, not only in 
Europe, but also at a world level, promoling con- 
tacts with North America, Eastern Europe, and 
the Third World: As for. the second, a cursory 


Blance-at ‘ables. _and 6 to contrast-the respective 


8 experimental papers 
for example, earned only 19.5 percent of the world 
citation total in 1964, a factor of four less than the 


situation. The third effect the ir impact On univer- 
sity education, is harder to gauge: undoubtedly, 
there has been some impact, but whether this is 
@reater than would have been the case if the 


resources invested in CERN had ‘been spent. on 


ther areas of scientii-c research is impossible to 
Judge. A si 


Again, ‘many instances of spin-off have clearly 
ansen, particularly t to -firms supplying equipment 


to CERN. as Schmied [60} has amply documented: 


However, “opportunity costs” have also to be 


technological spin-off higher. than it would have 
been if the resources spent on CERN had instead 
been used to support some other type of research, 
such as exploration of the ocean bed, for 


cxaonple? © To this question, there are as yei no 


we do not feel our assessment is intrinsically 


weakened by the fact that it has focused almost 
exclusively on — 


CERN'G mai purpose which isto provide Europe's ace: 


tists’ with excellent Facilities for high-energy physics re- 
Sagoo ac aap aap coe gee orate 
from the tweive Member States » belief ii 
ee a aes Pp. 2h epi ed 


which significant contributions to the advance of 
scientific knowledge have been made by experi- 
mental high-energy physicists using the CERN 
research 

“Three main sets of conclusions can be framed 


on the basis of the assessment outlined above and 
in-Paper I [51]. The first is_that, since about 1970 
(and_with the possible exception of a few years in 
the mid-1970s when. the Fermilab results were 
having a. major impact), the. overall record of the 
CERN i 


tory in the world | when judged in terms of experi- 
ments producing | Precise measurements and results 
with high s . 3 : 

| citations (which give 
some indication of the overall i impact of published 


st eis caked duescsibe’ of Wis Sake lide cd 
Martin (44), : 

“ Iv is, however, isoncwarihy dai i the siditude survey (de: 
scribed below), nearly twice as many physicists disagreed 
with-the following statement-as agreed with it: “ The level of 
Fesources speni on high-energy. physics can 
justified by the technological spin-off ii generates.” 
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tion was held -by Srockiavaii. and betare aha 
(from. the--mid-1950s until the early 190s) by 


mance evaluated in relation to inputs (number of 
users, funding, etc.) = the record of each o} the 
three US National Laboratories, and the Stanford 
Linear Accelerator Center in particular, seems to 
have been significantly better than that of CERN. 
Supporting evidence for sucha conclusion it pro- 


vided by the figures in tables 8, 10 and 14 in Paper 


survey “ conducted among the high-energy physi- 
cists interviewed in the study, :he great majority 


(11. percent) agreed-on the whole with the state- 
men = 


exception of neutral currents, the crucial dis- 
coveries in high-energy physics. between 1961 and 
1982° were all made at laboratories other than 
CERN: As the Director for International Co-oper- 
ation at the German Federal Ministry for Re- 
search and Technology 1s reported as Saying in 
1980, . 
CERN has been better at building wuperb acceleraion than 
a1 discovering spectacular physics (Dr. G. Lehr, reported in 
Walgate (69, pep 
feature of CERN's performance waa cleazly 
recognized: by high-energy physicists whom we 
interviewed in 1981 and 1982. Of these, 72 percent 
agreed with the statement, “The CERN accelera- 
tors have been responsible for a relatively small 
fumber. of major discoveries compared with the 
other main high-energy physics centres,” well over 
three times the number who disagreed (22 percent). 
en at CERN, this relative failure did not go 
iced. In 1980, the Research Director-General 


“ The altitude survey consisted of approximately 30 state- 
ments. relating (0 issues previously discussed in the inte: 
view. Interviewees had to circle a number on 9 ?-point scale 
depending on whether they agreed strongly (t): agreed (2); 
agreed but with reservations (3); were neutral or-had-mixed 
views (4); and 80 on up to (7), disagreed strongly with the 
statement. 
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te ‘the last few jears. the! most important developments i in 


sytumetry and.connected with the-4th and-Sth_quarks. Here 
the rok of CERN has been very modest ai compared with 
the discovery of the 3/pei at Brookhaven and at SLAC in 


1974. the discovery of the first charmed mesons at SLAC in 


1976, and the discovery.of the upsilon at Fermilab in (977 
(Van Hove and Jacob (67. p. 33)... _ 


pardcincas had come close to equalling these fig- 
ures. or having: an equivalent impact_on the ad- 
wance of knowledge. However, the ‘situation may 
have since changed radically with the publication 
in 1983 of the papers reporting the first observa- 
tions of the W and the Z particles ~ likely to be 
Seen £6 SC ol ie mos sporinm die rea oF 


ee Gees aan eos aan as a 


ven AGS accelerator can, as we have seen, 


be explained largely in terms of factors over which 
CERN itself had little influence - in particular, 
the relative lack of experience of PS users, the time 
ifcvilably taken by a large multinational organiza- 
tion in evolving efficient managerial structures and 
Procedures, “and. the traditionally. hierarchical 


Structure of European research activity. However, 
in the case of the  Intersecting Storage Rings, which 


jecisions | 
cara a aad Of tad Vamsel Gea ths cana 
ences with the ISR have been absorbed at CERN, 
evidence for such a learning process having taken 
Place coming from the very rapid and hugely 
successful exploitation of the protoo-antiproton 
Collider. As_for the relative lack of success of the 
Super Proton..Synchrotron in making major dis- 
coveries, the crucial factor was the four-year lead 
held by Fermilab. This cannot be regarded as 


‘entirely the fault of European politicians. If the 


Original design for the accelerator produced by 


CERN had been Icss expensive, then the American 


lead would probably have been far shorter. 
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ne the “crucial policy question raised -by 
the above analysis is the extent to which the 
factors that have limited the scientific Performance 
of the CERN accelera 
ari 


of this sort. Isi it -inevitadie > that a 1 laboratory where 
decisions tend to-be- made-by formal committees 
viduals or smal) informal advisory 

gr and which must be accountable to @ dazén 
political masters, will take less gambles and “sks 
than a laboratory free from such contstrainis? 
Most high-energy physicists interviewed did seem 
to feel that_“The system by which the allocation 
of time on CERN accelerators is decided, tends to 
encourage too much routine research rather then 
i ive but risky experiments” (51 per- 
greed . wi i 
Survey, appreciably more than the 3] percent who 
Sat ee The question | is whether, an intcma- 


ile } attempt to answer this ques- 
tion here, it is necessary to note 1wo Points: First, 
the decision in 1978 to. proceed with ¢ Sastruction 
of the proton-antiproton. collider did Fepresent: a 
major gamble by CERN, showing that the con- 
Servative tendency can be successfully resisted. 
Second, it would seem that high-energy 


the: United States is 


CERN. In the pasi, the situation in the large us 
laboratories 


a for five. years only -and,. excepi in one 
case, were not réappointed.) However, there has 
been a  Bradual contraction in the 1 number of Us 


may be more than the us oil oe abs to LS 
coming years “*), T?.. as that an increasingly 
Jarge number of scicnti.:; and institutional inter- 


© ‘See the pessimistic discussion of the financial future for US 
high-energy physics in Trilling {64}. 


ests have to be accommodated (and repre ed) 
w2thin the decision-making structure at cach centre. 
U: th: United States proves moore adept in - 
avoiding the dangers of. what many high-energy 
physicists termed “committee science” than CERN 
was in the past, “ then the previous ious ad ntage of 
US physicists over the rest of t ut 


penne tes optimism ‘in the longer term forms the 
subject of Paper III [52]. 
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search [CERN) az the period tee ta nd ee 
future prospects fo. CERN and its users during the next ten tc 
fifteen years.-- Se 

‘We concerned cursclves io the first paper (Paper i-~ Martin 
and Irvine [29}) with the position of the CERN accelerators in 
world: high-energy physics relative “to those” a! other large 
laboratories working in the field. We dealt primarily with the 
enad. from .1969 wo 1978, and atiempted wo establish bow the 
experimental output from the three principal CERN. accelera- 


pron is of ee Ba Seer nlc a3 
-_ In contrast, the second paper (Paper If ~ Irvine and Martin 
Yel eae pkg agg 


; (ous et pal oD PE 
how great have been the experimental achievements of ibe 
Intersecting Siorage Rings in world terms? And, third. how do 


pore petciemaaeac Owen Lock, for providing 
useful itical’ comment on an earlier draft of this paper. 
. the conclusions remain the responsibility of the 


Research Policy 13 (1984) 311-342 
North-Holland 


the outputs from the CERN 400 GeV pete 


ston and from a rival US machine ai Fermi-Navonal Accelera- 


"These Sectors are al relevance io ihe subject of ikis ibird 


apo spaiepemererent prepara 


in-other words, what is its Potential for fuiure development? 
‘The paper concludes with a discussion of the extent 10 which 


policy-making that such U techniques mugh: help address. 


= e overriding. goal of the various studies of 
basic science specialties that we have undertaken 
wyilaiati suede lee sakatig oy ae 


(23). 
and electron ‘ca physics (Martin and Irvine (27)). 


0048-7333 /84/$3.00 © 1984, Elsevier Science Publishers B.V. (North-Holland) 
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proach the end of iheir useful tesearch “ifetamies. 


ment for improved -methods for a 


future success of new research facilities. 


The need for predictive. ve Boticy toot | is greatest 


volved are now of such a magnitude tha 

sions of them are central in the formulation | of 
. Qverall national science policies. The cost of the 
planned ‘new LEP accelerator at CERN (over 900 
million Swiss francs).is. for example, appreciably 
greater than | the: total British. Science and En- 


The emergence. of zach aguas in ihe 
peer-review p, oveés (the method traditionally used 
in -allocating- resources for -basic science) has in- 


creased the need-for other sources of information 
to: aid policy- 


mak 
ly 
distribution of funds- -between- specialties | as. they 


fespect, we would argue that systematic compari- 
sons of past. scientific_performance (of the sort 
reported in. Papers 1 [29] and II_[24}) can be an 
important input into decision-making. particularly 
when genérated- by: analysts. outside the: social 


ral science in 1983/4. At — time when thee are 


severe pressures on funding, it is important that 


Science. specialties are still, able- to make_ invest- 
men ment decisions solely. or even largely, on the basis 
Of perceived scientific merit (cf. Irvine and Martin 
(26). As AS scientific activity in. such specialties is 
concentrated in ever fewer_research centres (for 
example. in the United: States there are in 1984 


munity concerned... Wheri. resources. are heavily 
concentrated on a single centre or research facility, 


telescope), it is desirable to now | eet successfully 
the previous research facilities have been operated, 
in particular compared to similar facilities at rival 
centres. “External” evaluations are probably more 
likely to: be trusted by sections of the scientific 


only four sree high-energy physics centres, 


to exploi 
underlying matey is that it is age more 
difficult in Big Science to locate neutral peers 
capable of providing sufficiently disinterested 
judgements: al) potential peers tend either to have 
Some professional interest in_a proposed new pro- 
ject, or to be-associated with a competing set of 


2 For-example, Britain spent £832 milion in supporting 
baiic research in astronomy. spece. and nuclear physics in 
1981/82. This compared with £21.8 million spent by the 


Science-and Engineering Research Council. on -peer-re- - 


viewed granis lor all other areas of natural science. £20.7 


lion by the 2 Agricultural § Research n Council, £ £543 million by 
the Natural Environmem Research Council, and £101.7 
million by-the Medical Research Council (sce Irvine and 
Mantin [26)). 


community outside: the: Big Science concerned concerne 4 


while they may. acceptthe need for-major-invest- 
ment in Capital-iniensive research facilities, they 


St mocarmenite naa 


~- ‘This said, it isc clear that "track record” is only 


predict likely future- research _performance.~-Al- 
though perhaps marginally better than no fore- 
casts at all, predictions of the future based upon 
simple extrapolations from the past are unlikely to 
be” “petticalariy successful (see. Miles: and Irvine 
2 characterized 


by rapid change. This is not to say: that extrapola- 


they aré more likely to have some validity when 
based upon an understanding of the factors that 
have structured past performance, and of the ex- 


tent to which they are likely to affect future per- 


formance. For this reason, an attempt has been 


made in the present study to extend the previous 
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evaluation meihodology: (which was concerned with 


ities — see footnote 1) to include an 


A range of information has thus been. obtained 


Concerning the factors structuring success and 
failure in the. past - for example, whether there 
has been a strong user-group associated with the 


accelerator, whether the scientific management of 
the facility has been effective, or whether from-a 


technical viewpoint the accelerator and subsidiary 
instrumentation have been of a sufficiently high 
quality. - 


Yet-even assuming that these factors continue 
to be-important. this is still an insufficient basis of 
which to make decisions concerning a major new 


facility. What is. needed i in addition is information 


facility and the research to be. carried out on it - 
the “ripeness” of the research area, ihe relation of 
the instrument tO other Facilities (for instance, 
whether it. duplicates, or is complementary tc 
Facilities Glsewhere), the degree of 1 


form poy of instrumental pened that can 
be applied in other research activities. = 

-_7In this way, then, it is possible to envisage the 
formulation of a set_of criteria to help assess the 
Ment of proposals for future research facilities. 
Certainly. — of score are.already used 
. However, 
sich cfiteria ate not aaayi made publicly explicit, 
and the information required to utilize th aa 


efforts (and a- resulting formation of | strong inter- 


est-groups focused on each of the main centres in 


process may be reduced to little more than 2 battle 
between. institutional interests “(cf._ Irvine. and 
Martin [26]) The-nderlying problem is that, in the 
present system of decision-making, theze are inad- 
mechanisms for ensuring accountability to 
tside the Big Science concerned. even to 
ientisis in neighbouring specialties. The aim in 


. increase (in real t 


‘fovpepedeibiuscs ie ea ies for devel: 
oping a framework to be used ia anal ne (ire 
policy options in a sysiemat 
ble: way. If the scope Tor utional iobbying is 
to be limited, then deliberations over costly new 


research instruments like accelerators or telescopes 
Ore ers een i 8 ier body of 


© 


move ultimate responsibility for the distribution of 
resources for research from scientists. but merely 
ensure that’ scientific decision-making was con- 
ducted in a more public arena. we 

--4n_this paper, we attempt to “assess the: future 
scientific Prospects for CERN: In particular, we 
consider whether the set of criteria described be- 
low“can aid’ analysis of the likely prospects -for 
CERN's. major new accelerator, LEP, relative-to 


ous construction. projects in terms of on i 
financial commitments it implies for CERN Mem- 
ber States and its. delimitation of future Options 
for the laboratory. -Whereas previous accelerate or 
construction programmes at CERN have been es- 

sentially discrete efforts with a time-hc 


project is differe si 
Ps of he projet (orach a heamene of 


50 GeV) scheduled for completion in the second 
half of 3988, subsequent phases to reach first 


approximately 90 ¢ GeV and then 130 GeV are also 


complete. Formal approval of ‘these later phases 

will obviously not be obtained for some time. but 

there must be confidence at. CERN that, onc 

Phase L andthe 27-kilometre tunnel it 1 es 
ted, the a rt 


) in the CERN- budget. At 
such a rate of funding, a 130 GeV-ve version of LEP 
would probably r Not come into Operation until the 
early 1990s, i.e. nearly ten yes: $ from now. 

_"At that stage, there are likely to be strong 
afguments that, since the 27-kilometre tunel al- 
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ready « exists, it would be relatively cheap. 10. cone 
icting proton synchrotron of 
perhaps up to 10,000 GeV (10 TeV) by placing it 


alongside LEP. Such a machine could take another 


Eve Jenrs oF 8910 constTass Moreover. once com- 


years 10 bring to fruition. In. this-way, LEP will 
provide CERN i with. a succession of possible new 


implications for what CERN is doing twenty years 


or more-from pow, it is surprising that there was 
i of 


en “in 1981. Certainly, « 


and: 

Search- nde agencies in the Member States. 

However, these discussions were circumscribed by 
the decision of CERN management that LEP 
should be. treated not as.a “new” accelerator but 
as an extension of existing CERN - facilities, a 
decision iaken to minimize the risk thai political 
debate might delay the project's start. LEP may 
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technica] requirements, and attempts to assess the 
extent -to which these are likely to affect. their 


(and, if so, for how: long), the likely capacity of 
that accelerator to generate new physics results, 
and the ability of its user-community to exploit the 
Scientific potential ‘of. the new facility to the full. 
The third and fina] set of comparisons centres on 
the spin-olf, or contribution to accelerator physics 
and technology. likely to be generated by each 
machine, and on its potential for development at a 
later stage to yield a new research facility. The 


methodological approach described below for sci- 


ence-policy purposes, particularly in those fields of 
science characterized by heavy capital ana 


with "Sadie science-policy sages this _at- 
tempt at constructing a framework for systemati- 


cally assessing future s should be regarded 


largely. around a single laboratory forms an ade- 
Quate basis for. committing substantial long-term 
resources - particularly when the funding involved 
is of such a magnitude that it cannot fail to have 
¢ Other scientific specialties. 

In analyzing whether improved procedures can 
be developed for assessing the future prospects of 
large capital projects. we shall first examine the 


history of the LEP project. This is followed by a 


brief review of the other main accelerator (or 
collider?) projects planned or, underway around 


jn-this and Papers I [29] and Ii [24], the term “accelerator” 
is sometimes used generically to cover both accelerators and 
colliders. 


evaluations. The intention-is that such assessment 
excercises be undertaken prio to ‘major research 
investment decisions, and then repeated periodi- 
cally during the construction and- operation of a 
facility. Kh should 10 also » be Stressed é that studies such 


“date” quite rapidly as they are: overiaken by. 


unforseen events. Nevertheless, while questions of 


‘its , publication and circulation, we believe most of 


the main issues and trends will not. To this extent, 
there is therefore an clement of “testability” about 
the validity of the approach that we have adopted. 
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the previous twenty years had come from proton another mouable sdvance within high-energy 
accelerators ~ generally from the highest-energy physics at about the same tirme that came-1o ex- 


machine operating at the time. Even in 1970, most 
senior high-energy physicists saw no obvious rea 
son why this-trend should not continue, and the 


thoughts of many of the more ambitious accelera- 


ry unifying the four 
- in natuve has bee 


mesons, and $0 on ~ 
that until then had eluded the much higher-energy 
Proton machines. One fairly immediate conse- 
uence was the race between the German labora- 
tory. DESY (the largest of the European electron ; 
accelerator centres) and Stanford to construct a 3 Though the Sinolord ciectran-pasitress collider i anand 

boca Bae - PEP. the first “p” in the acronym originally stood for 
* ‘This and the other dramatic discoveries made in ihe sabie- “proton”. See Metz (30, p.853] for details of this earlier 


quent three years are described in detail in Paper II [24]. Project. 


o.. 
ERIC 


248 


ne B.R Marin ond J. Irine / CERN: 


Proton collider. $ Since the history of this project is 


closely interwoven with that of LEP, it is worth 
looks i 


During the mid-1970s, concern was being ex- 


pressed within CERN. that the SPS was coming 


identical Fermilab accelerator (cf.-Van: Hove [41, 
p-31})-. Hence, when the Italian physicist, C. Rub- 
bia. began to advocate the conversion of the SPS 
(and the Fermilab accelerator) into a proton-anti- 
proton. collider capable of reaching the_ inter- 


Past performance and fotere prospects. Ww 


__The main uncertainty associated with the pro- 


x 
the factors: determining the number of particle 


collisions per second) to make the physics -accessi- 
ble. However. the results of various tests of the 
technique of stochastic cooling (pioneered: at 
CERN) between 1976 and 1978 were sufficently 
encouraging 10 suggest that a lumiposity. of 10 
cm? ¢—" (the eventual design figure) might be 
achievable. - sufficient to generate tens of Z, 


mediate vector boson. region. be found a receptive 
ence within the European high-energy physics 
community. A further incentive to procced with 


the project came with the realization that the-SPS 
—— that it could pe converted 


Fermilab: accelerate. giving p= hysicists 
the chancy of ues ibe wiles aad pocieg several 
factor that may have contributed to the attractive- 
ness of this: project concemed its role in the 
longer-term. budgetary plans of CERN. Jo the 
absence of a major capital-consiruction ion project to 
fill the hiatus between the completion of the SPS 
in 1976 and commencing the building of LEP in 
the early 1980s, CERN would have found it politi- 
cally difficult, if not impossible, a kegel its 


ee a ee 22 ee = 


assumed that-this would still be high - ai to 


permit the new particles to be discovered some 


ton-protoa collider then scheduled to begin oper- 


was not without its opponents; some users of the 
SPS accelerator in particular were alarmed that the 
conversion_of the machine into a collider would 
Cause the loss of up toa year from the fixed-target 
experimental _ programme (which - was _ already 
labouring under the handicap of having. siaried 
over four. years behind: that of Fermilab), that it 
might affect the reliability of the original synchro- 
as it in fact did), and that, when completed, 


colli igni: proportion 
of the SPS’s time, leaving insufficient for fixed- 


target-_ experiments. Nevertheless, the CERN 


) MSF required f for experimental Opera- 
tion and routine investment purposes. Once re- 
duced to the “base Joad” figure, it might have 
proved even harder for CERN: to convince all the 


needed to be substantially increased again in order 
iO begin ‘the consiruction of LEP: ° ‘The 
proton-antiproton collider, with. its relatively 
modest cost (3) MSF) for a new -high-ene 


* Cenuinly. CERN | 


argue that LEP constituted no more than an™ extension” to © 


the existing experimental programme (a point discussed 
Further below). piesa Serie 
7 Similarly. one of the main arguments for proceeding with 
the construction of the ISR in the late 1960s (according to 
senior physicists interviewed a1 CERN) had been to fill the 
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management decided in 1978 that this major 
sacrifice of prime research time was a price worth 
paying in return for the chance of discovering the 
all-important intermediate vector bosons. Once the 
go-ahead had been given, contruction of the col- 
lider proceeded rapidly, the first collisions being 
observed_in July1981, and,.as we.saw in Paper I} 


[24], the W and Z intermediate vector bosons were 
discovered amid much publicity.in 1983... 

In the meantime, there had been significant 
progress with the LEP project. Between 1975 and 


experimental needs in ihe period up to eds of 
this century. Among the options considered_for 
CERN were a very high energy fixed-target proton 


“gap” between. refurbishing the PS and beginning work on 
the politically delayed, but higher priority. SPS. 
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synchrotron, a proton-Proion collider; an_élec- 
proton collider, and-a large electron-positra.- 
ider, LEP. The European Committee for Fu- 
ture Accelerators (ECFA) played a prominent ro!- 
in this discussion (much of it was carried oui in 
the: closed sessions - of “the “Restricted ECFA“), 
and €véfitually in-1977: came. down Strongly in 
favour of LEP. Their decision. appears to -have 
been based-on a widely accepted set of scientific 


“new physics” of quarks-and leptons, and the 
prediction by a large-body of theorists that the 
energy of the Z; was such that LEP would be the 
ideal- research instrument on which lo mounta 
Systematic investigation of the particle's properties 
(its. width, decay-channels, and £60.00), Another 
Strongly influential factor was the fiotion of com- 
plementarity: Fermilab and Serpukhov were plan- 
ning proton synchrotrons of 1 TeV and 3. TeV 
resegteael and Brookbaven was about to- em- 


impact of both the PS-and SPS (see Paper II {24]). 
A fina) factor that appears to have influenced the 


pent ~ a feature of the decision we shall 


discuss later... == = 
By. 1978. the CERN Courier 1 Pp 
lo report that, after much discuss 


ah was able 


rt ronsensus mar “come. Out strongly for: ‘an 
electron-positron machine to take colliding beam energies 


well beyond those which are accessible with PETRA at 
DESY and PEP at Stanford. 


al parameters of the machine had 
also been determined by this time (in particular, a 
ring of 30 kilometres in circumference had been 
agreed, rather than the $0-kilomeire and 20-kilo- 
Metre options considered previously), while cost 
estimates were calculated shortly afterwards By 


adopting a “missing-cavity design" (in which ra- 
dio-frequency cavities were Progressively added), it 
was proposed to build LEP in severat stages, re- 


aching a beam-energy of 50 GeV for a cost of just 


Over. 900 MSF. Energies of- 65 Gev. naa 90-GeV 
would subsequently be achieved: for-a total cost of 
1065 MSF and 1275 MSF respectively (cf. (8, 


11). The fi 


the conventional, sl eadio-Sreqweney: cavities, with dane’ 


new project is one thing: guuaag G2 tek aa 
Political agreement from. Member States to go 
ahesd with the construction programme is another, 
aS_we saw in the discussion of the history of the 
SPS project (see Paper 1 [29]). The SPS was severely 
delayed because of lack of agreement over both 
finances (with Britain threatening withdrawal) and 
the site of the new accelerator, According to senior 
physicists whom we interviewed, similar political 


eateea ee 
soon became apparent from preliminary discus- 
sions with. funding bodies inthe various Member 
States_that it. would be politically. difficult, if ; not 
impossible, to finance-the-new accelerator by an 
oe oy budget. With: the 


States that the new machine-be financed from the 
existing CERN budget (some 600 MSF per annum 
at -1978 prices), even- though this would entail 


Closing the ISR ° and perhaps the small synchro- 
cyclotron, and curtailing the SPS experimeénial 


led to propose to Member 


‘ Robinson ps. p.531) has quoted: an approximate gute or 
by superconducting elements and for construction of the 


four remaining experimental areas. This project is referred 
: to below as LEP-130. 
resulted tn & vocal-compaign by s. ‘Segnificant 
Minority of CERN physicists agains: closure of the ISR. 
Arguing that the machine bad a good track record and was 


still a unique facility RoE pimp pp opera 


pagar ine te eh pase gate 
period, thus freeing resources for continuing the ISR pro- 
Eramme. Some sdmitied that closure of the ISR was the 
Political sacrifice that probably had to be made. to guaran: 
tee the future for LEP, but were bitter about the lack of 
Public discussion on the matter. In a similar vein, the 
Scandinavian countries rapidly mobilized against the possi- 


955 
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programme: This also involved Gimming the Zosts 
of LEP. in particular by using the SPS io inject 
electrons and positrons into the main nng rather 
than. building a new 22 GeV electron synchrotron. 
Although this further reduced the SPS pi 


ties™ {9. p.192}. to be absorbed ander the “basic 
Programme” of CERN, rather than as anew accel- 
erator '° requiring the authorization by Member 
States of a supplementary programme — authoriza- 
tion which might not be immediately forthcoming. 
Third: in response ¢ to worries by the smaller. coun- 
tries about possible cost over-runs in the LEP 
project as a result of technical probtems, '' CERN 


ya (cf. 
a. eon This means a any unforseen diffi- 


bie ciosure of the_synchro-cycloiron (which many of their 
proysiis ve wel and eventually obtained agreement for the 

of the experimental programme on this scce!- 

- erator ona ' Eniied bass 


‘0 ‘This would seem to be a somewhat artificial distinction and 
one that might prove rather har3 to defend if subjected:to 


for are any Pleaned: the ISR and SPS were merely “exten- 
sions” of the CERN PS,-SPEAR and PEP-were extensions 
to the Stanford linear accelerator, ISABELLE would have 
been just an extension of the Brookhaven AGS, and so on. 
‘While this reclassification of the status of LEP may have 
been admirably suited. to CERN’s purposes in that it greatly 
reduced the risk of political disagreement as among, Member 


ber Sistes a o segasde diacoeioes of other types of cw 


facility. It: might-have been expected that such a termino- 
logica) “sleight of hand™ would give rise to some con- 
Uroversy about high-energy physicists’ arrogation of govern- 
ment funds while seeking 10 limit the degree of public 


-- accountability, but none seems io have arisen. - 
‘| Problems. were, for example. anticipated with: sunnedling 
under difficult rock conditions in the Jura mountains. As 3 


result of testborings. the-precise location-of the LEP tunnel 


had to be changed: the.circumference of the accelerator was 
decreased from 30 to 27 kilometres and the length of the 
tunnel under the mountains reduced from 12 Mlomerres 0 


sarine G Gast nie ce ae 
of the first experiments on LEP or Curiailing even 


Se Pe ane een eee oe oe 


agreed to proceed with the project. As matters 
stand at the time of writing. it is anticipated that 
Phase | (to achieve a beam-energy of 50:GeV and 
to construct:four out of the eventual eight experi- 
: I halls) will be pleted by the secood hal 
6f 1988 (cf. 22, p-228}). The estimated cost in 1981 
prices is 910 MSF; to this must be added between 


CERN). 


3. Likely competitors to LEP 


_Belote we move on in subsequent sections to 
the « © advantages. and. disad- 


These competitor machines have been classified 
into four categories: (1) fixed-target proton syn- 
chrotrons; {2) proton colliders; (3) electran=pro- 
ton colliders; and (4) electron-positron colliders. 


3.1. Fixed-target proton synchrotrons 
_ Deiails are given in table: of the three planned 


(of p«:sible) major new accelerators which may be 
joned before the end: of the century: For 


comparat - 
for the largest proton synchrotrons already operat- 


__ ing. 


3.1.1, 3 TeV UNK (Serpukhov, USSR) 
__-As was seen-in Paper Ii [24], , by the time the 
experimental programme of the Serpukhov 70 GeV 


accelerator was in full operation, the much 
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Table i Bee oe 
Major protor. synchrotrons = present and proposed * ad padi cia Ia 
Date of -Maumum — Centre-of- intensity Esumaied Penodof Future 
first, _—seam-energy = mass.energy (particles. cost -- world, options € 
expenments (GeV) (GeV) per pulse) (MSF") —ieadership 


* This lists only the world’s highesi-energy machines. 


» These are the approximate estimated costs in millions of Swiss | tranes bated on 1983 exchange j rates. (An exchange rate of S) =] 


_ Soviet rouble has been used.) Ss 


5 Be= prowon- proton collider: pi = proiaéi-antiproion collider: ep-=electron- proton collider. 
* This figure is rarely achieved. The “normal” maximum energy is 400-450 GeV. 


* Experiments at an energy of about 800 GeV are planned for 1984. 


ligeerssagy 3 mice. at_ Fermilab was nearing 
completion. Once their world lead in energy disap- 


peared in 1972. Soviet high-energy Physicists be- 
gan planning for the construction of a major new 
proton synchrotron, UNK, the energy of which 
was originally set-at 2000 GeV (2 TeV) 
reported in-1976 thai the Soviet Government had 
Pledged 200 million roubles (about $200- million) 
to the project [4. p.401]. By 1980, the final design 
parameters had been settled _the proposed 
originally esti- 
mated that the machine would take approximately 
seven years to complete (Wilson [43, p.40}) from 
the time_constructi ction’ began in 1981: althougt: 
1990 is now considered to be a more realistic 
completion date (cf. [45, py 


3121 TeV. Doubler / Saver (Fermilab, USA) 
Shortly after completing its 400-500 GeV accel- 
erator in-1972, Fermilab began work on the En- 
érgy. Doubler project to increase 
GeV (1. TeV) by installing supercon ig Mag: 
nets. As with LEP, the haay, of decision-making 
over this new accel 


o) ally classified by Fermilab as 
nal. R&D programme, for which r no'special 


a The existing 70 Gev accelerator. suitably, upgraded, will be 
used ject protons into # conventional 400 GeV booster 
synchrotron, and subsequently into a 3 TeV superconduct- 
ing-magnet proton synchrotron (cf. (11. p.147)). 


States Office of Mana- 
panen and Budget insisted that the work had 10 
} @$ & construction project. Since the 
mary for obtaining authorization for provedis 
costing under $50 million is considerably less severe 
than for those requiring higher expendituies, cf- 
forts were made 10 keep the estimated costs below 
this limit. When it eventually became clear that 
this was -no longer feasible (in part becar 
greater. than’ expected technical 
superconducting magnets), the 
into two. pas: (1) the 


to make possible fixed-target experiments at_an 
energy of 1 TeV, play elapetg tye ranteoe 


rose sbove the $50 iniltion mark,_this project too 
was divided “into_two ‘paris ~ Tevatron I aud 
Tevatron II. By 1981, the estimated costs of con- 


TeV accelerator. the Depariment of Energy (ihe main fund: 
ing body in the US for high-energy physics) wai more 
concerned that the laboratory shouid make effor 
duce its elecuricity. bill, a-goal tow. 
tng magnets would coninibute: hence the alternative pi aa 
tide. the Energy Saver. 
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Tev fined target accelerator 


(Tevatron 11) had-risen 10 between $110 and $120 


million.'*- to which Pur be- added the cost of 


the higher-energy eosices. Experiments at_an 
energy of berween 800 and 900 GeV are to be run 
in 1984, with the. maximum energy being achieved 


probably then remain the- world’s highest-energy 


accelerator for the rest of the 1980s. 

z I: Zz 10 TeV. Superconducting Synchroton (CERN) 
. The 27-kilomeire tunnel for LEP is. designed to 

enough 10 accommodate one or more 

additional rings of magnets. Consideration has 

already been given at CERN to the possibility of 


ing a ring of high-field nipeiooseatng 
ble 


1990s): to yield a proton potee 


with a maximum energy of approximately 10 TeV 


(cf. Wojcicki [44, p.26}). 


3.2. Proton colliders 


(and ania) ss colliders likely to-become opera- 
tional over the next decade: as -before,- the- ‘table 


proposed 
Cintas: Beam Accelerator. ayer “called 
ISABELLE) is_included, even though this project 
was halied in 1983, the reason being to ascertain 
the oe for this collider had it been 


3.2.1. -400- GeV - Coitiding~ Beam Accelerat / 
TSABELLE (Brookhaven, USA). -.. ~~ 

When Brookhaven completed: ihe AGS Conver. 
sion Programme (sce Paper II [24]) in 1971, the 
laboratory began to plan a superconducting pro- 
ton—proton collider with a beam-energy of 200 
GeV. Early tests wit Pr I ct- 


| type Supercond: 
ing magnets proved very encouraging; '’ the field- 


Figures provided to the authors during a site-visit to 
Fermilab in_late 1981. 

Sas early as 1973, it was reported that, “ The performance of 
the [prototypes} seems to have answered several outstanding 
Questions concerning superconducting magnets. One con- 
cerns the reprodiscibility of magness.” [2, p.374]. In the light 
ofthe major problems with magnets experienced later 
(dese ad ten). this conclusion seem: to have been a 
lade premature. 


la required for a 200 Gev machine 
was easily attained. and. by 1976. a figure of 5 


tesla seemed to be within teach. The Ca High 


data from even higher energies, recommended that 
the planned beam-energy be increased 10 400 GeV. 
With hindsight. this decision is now regarded by 
many. physicists as a mistake. Although an_in- 


dustrial contract. was awarded in 1978 for winding 
the magnet coils. by 1980 it- = that the 


sen by. Brookhaven was incapable of achieving the 
necessary: field-strength. In 1981. after cons.der- 
able -internal - organizational upheaval ct the 

be 


sulted not only in a dramatic escelation in_ the 
estimated costs of ISABELLE, ‘as it was then 
Called; but also in the likely completion date being 
pushed ever further into-the future. In 1976, the 
Cosi of the 200 GeV. collider was estimated as $166 
million, and a year later, after the energy had been 
increased to 400 Gev, a figure of $245 million was 


had already been spent (el Broad [1, p-1089 

1982, it was clear that the collider would not: be 
completed unti] 1988 and ‘that a further $500 
Milin7:. would be required (cf. Trilling (40, p.26 
9}). In. view “of the parious state of US 


ee ee ee 


- options were found tc _:.udest and were deemed 


insufficient to justify the significant. reduction: in 
physics potential associated with them. As a result, 
the Brookhaven management decided in February 
1983 to persevere with the original plans for a 
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Table? 
Mayer proton colliders - present and proposed 


Date of Maximum 
first _. beanr-energy 
experiments (GeV) 
CERN ISR 1971 u 
CERN pp 19Rt ™ 
1000 
ri 
40 + 3000 
_ ducting Collider? (99th? 10,900 


Luminosity Estumated Num‘ cr of Petind of Future 
tem 72. ') om - ceperimental world ophons 
(MSF *) arcas wadership 
= (years) 
Sa ee ee 
fo” - 1 - 
to%'7* 204 2 $ 
ion +350" 2 ~ sr - 
10 or 0 in cl 
- 33 ion.cotlider. 
x 3000 + 3000 GeV 
-10" ’ ” 2 Dp and pp 
~10"7 ~ 900" 6 ? 
. 2 ’ ” op 


Hy (noedroad am{ puo suomuofaed ig ‘NYT / peru, ifipwo wow! wg 
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Proton-proton collider (iow renamed ihe Collid- 


construction schedule enabling -the first. colliding 
beams to be obtained in October 1987 was- drawn 
up (cf. Wojcicki (44, p.31}). However, in July 1983, 
HEPAP recommended that the CBA be discon- 


4, p.7}) and the project was finally 
the Department of Energy a few months 
later. By then, a total of some $200 million had 
been spent on the accelerator. 


322, 1 TeV jroton=antiproton collider ~ Tzairon 


value of some $40 million to an estimated $70 
million in 198}. In addition, the-decision in 1982 
to develop a iiew and more efficient antiproton 
Source was reported as increasing the cost by a 
further $40. million. To this must be added the 
outlay on-the 1wo detectors, giving an overall total 
of.around $170 million. " The first proton—anti- 
proton experiments are planned for-1986, which 
will give CERN. five years lead with its (lower 


energy) collider ~ the same advantage previously 
enjoyed by Fermilab with respect to the 400 GeV 
Proton synchrotrons. 
5.2.3. Serpukhou collider (USSR) cane 
:. Once the 3 TeV proton synchrotron (UNK) has 
been completed at Serpukhov, it is intended -to 
collide 3 TeV protons with 400 GeV protons from 
the booster synchrotron (see-footnote 12- above), 
Biving a centre-of-mass energy of approximately 
2200. GeV. There are also preliminary plans for 
phase I-of UNK which include 3+3 TeV pro- 
ton-antiproton colliding beams and, with the ad- 
dition of a second superconducting-magnet ring, 
3 + 3 TeV proton—proton ceilisions. 
5.2.4 2 TeV Dedicated Proton—Antiproton Collider 
(Fermilab, USA) ee ee oe 
In 1983, Fermilab pus forward a proposal to 


"* Robinson [39, p.16) quoved a figure of 3569 mill’. ia 
1982 for the combined cosi of ths 1 TeV accelerator fin- 
cluding its upgraded experimental areas) and the coluder 


build a-proton-antiproton colliding facility (called 
the: Dedicated Collider) with planned (beami-en- 
ergies of up to 2 TeV and a design luminosity. of 
over 10°! cm=? s~". Although it would age the 1 
TeV Tevatron as an injector, it would have. the 


As already mentioned, the tunnel for LEP.can 
accommodate onéor more additional: rings of 
magne ts. By installing a ring of high-field super- 


~_The year 1983 marked something of a turning 
point in US particle physics: After strong pressure 
from.the Administration, the High Energy Physics 
Advitory Panel (HEPAP) was finally able io reach 
a consénsiis agreement that the Colliding Beam 


orksbop held 


at Comell University, HEPAP vnanimously en- 
* dorsing in July 1983-(three months later) a secom- 
mendation by its Sub-panel on New Facilities for 
ihe immediate initiation of the project.-Provisional 
Geir aateS suggest that the construction of such a 
Kwchine together with the necessary preliminary 
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udy-b fi 
for the project. If the United States then-decides to 
Proceed further, the Department of Energy annual 
budget for high-energy physics would have to be 
approximately double the 1983 figure during the 


five to six years of construction. 


33. Electron proton colliders 


proton: or electron accelerator. -Untit _Tecently. 
however, these schemes hae rarely progressed be- 


yond the conceptual stage. Vable 3 lists details of 
the colliders in various stages of planning. 
TLL. 30 GeV electron and 820 GeV proton collider 
- HERA (DESY, West Germany), 
‘The West German national accelerator centre. 


pipeteee-ta aap ring. (PETRA) 0 yield colli- 
sions between beams of electrons at 20 GeV and 
protons at 280 GeV (ef. [5. p.364)). This scheme 


Past petfornomce ar rs feiare prospec um 


pases uircng support from the ae Com. 


miter (or Future Accelerators, who gvbogets that i ‘it 


poner 
yeras ahead. of a similar collider then being built 
by. their great. rivals at Stanford - adopted a more 


million pauline (some | $265 million or 530 
MSF) over a COnStTUCHOR < mid of seven 1 years 


- year 1980 also saw the a Federal 


Te for Research and Technology (BMFT) 


ee ali then = consideration in the 
country. The- Pinkau Committee, which Teported 


opt ropp a ng Sc 
way towards assuaging the fears of those who 


believed that the energy of the electron ring might 


"7 The electron ring would be! built first giving an. option tor 
carrying out 30 GeV eleciron-positron experiments in about 
1986. This-gave rise to fears-axvong certain LEP supporiers 

that ihe energy of this eleciron-posiiron collide: might be 

“stretched™ using superconducting radio-frequency cavities 
to achieve £0 GeV. and hence to reach "he Z, region a year 
or so ahead of LEP. 


Table 3 ae aeer 
Major electson-proton (ep) colliders = present and proposed: ns SSeS SEES = 
Date of Maximum Centre-ol- Luminosily Estimated Numberof Period_ Fuwre. 
first cam-energy massenergy (cm@7s"') cost expesimental of world options 
experimes.. (GeV) (Gev) (MSF 5 areas leadership 
es ss asec zens : 2 : (years) 
DESY_HERA _. 1990. W+8w 33 3x10” - 440" 4 $+? ? 
KEK TRISTAN = - 19907 25+ 300 1"s 2 2 ” 0 o] 
CERN ep‘? mid. 1305270 bs ? 1 ? ? 
1990s? 


* See note b io table 1. 


® This ix based on the most recent estimate of 660 million | Deutschmarks at 1981 prices (cl. (20: p90). 


* Thisp is project might involve colliding 130 GeV electrons from LEP with 270 GeV protons from the SPS. one of the possible long-term 


options for CERN. 
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be ened westigate the detailed properties 
Of the Z, particle abead of LEP (see footnote 17). 


PETRA). and: partly to give time for the further 


research and development work on the supercon- 
ducting magnets required for the proton ring. ** In 
1983.: BMFT_ decided to include in its planned 
budget for 1984 funds to enable construction of 
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laken unl the completion of LEP. (and tbe 


ing 20 GeV electrons with 3 TeV-protons could 
also not be realized until well into the 1990s. 


3.4, Eleciron-pasitron colliders 


: Because of the very great interest in asceriain- 


HERA to begin, provided that sufficient foreign 
participation-in the project could be attracted (cf. 
[20,p.90). It was estimated that construction 


\ - ; ia Jepanest National 
Lahorateny| for High-Energy Physics (KEK) has 
embarked upon the construction <” a 30 GeV 
electron-positron collider, TRISTAN, In a second 
stage of the project. it is plarncd to add a ring of 
4.5 tesla superconducting magnets capable of ac- 
celerating proinns ‘can exergy of 300 GeV. Colli- 
sions of the proton beam with 25 GeV electrons 
will - yield a centre-of-mass Heng flak of about 175 


(see above) incorporated an option Sor colliding 10 
or 20. GeV electrons with 2 TeV protons (cf. 


Sopp apenas Hence a great deal of 
effort has been put into planning the contruction 
of electron—positron colliders, both at CEP.N with 
LEP, and at a number of other laborsiories. 7 able 
4 lists the main details of LEP and its bkely 


: Capable of reaching a 
SnCoini merge at @O Gar I was, ontimsated 
that the project would take five yeass i conmplete 
and cost a total of 7.5 x 10°" yen Vecdoui S350 
million or 700 million Swim. frsscs) (2! [5% 5: 


National Laborstory for High Eat 


Wojcicki [44, p.33):-CERN and. Serpukhov also 
both havea future elects proton Option under 
consideration. At the former, electrons from LEP 
could be collided with protons from the SPS (cf. 
{8, p.5]). However, since it is now more difficult to 
obtain authorization for increasing the CTRN an- 
nual budget (in view of the prezs:..25 on sientific 
funding in most Member States), the construction 


of such 2 facility could probati; not be under- 


" Brookhaven's_ traumatic. experiences” with” ISABELLE 
showed all too clearly the dangers of proceeding with the 
construction of a new accelerator before all the necessary 


RAD on superconducting magnets had been completed. 
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tional radio-frequency cavities, hut supezconduct- 
ies ‘are being -develsjesnd ot- KEK -and 


elsewhere which could permit ihe cenucof-mess 
energy 10 be raised at = Hater stage te: approxi: 
mately 90 GeV, bringing 2, physics: within range 
(ef. [18]. in-addition, as we saw earlier, there are 
plans to add a proton ring, thereby generating 
electron-proton collisions. 


34. 2 50 GeV Stanford Linear Collider - SUC 
(SLAC, USA) Sp Moise esis fit atle fn 
In 1977, when the European Committer for 
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Table. 4 
Mawr electen— patron colliders — presen and prepased™ =e = = : 
Date of Maximum = Centre-of- Luminosity Estimated No.of  Penadof = Future. 
first beam-energs mass-energy (cm’*<"') cost - experimental world options 
experiments (GeV) (GeV} (MSF*) area leadershit. 
Ze 7 (veats) 
DESY PETRA 1998 igh 3 - 4 a a 
SLAC PEP | 1980 td 6 - 6 0 reg sicker ls 
KER TRISTAN 19%6 0 66 - 700 4 1 TRISTAN-45, 
. Pp 
SLAC Linear ae e _ - : : =e ae 
Collider (SLC) _ 19%7 Mw 100 6x10" 240* 1 ” 
J9RR ct) 100 3x10" R04 $ 
198K 3» 100 3x10" ~1200 4 -? 
CERN LEP-90 ~ 19907 90 1R0 10° ~3300 4 -7 
CERN LEP-130 —easiy 
> : 1907, 130 260 x - 4007' & 3-37 
Novosibink tg: __ is sie -- 
VLEPP - 1990s? 150 300 - 10" 7 § 3-5? 
SLAC Large Linear mid/laie -- 
~10" +0000 6 ? ? 


Collider? 1990s? 1000 2000 


* Sce note b to table}. - | 
” Ungraded to 20:GeV in eee cis 


* Pregeci discontinued in 1983. 
Furate Accelerators decided that/LEP should be 
CERN’S next major facility. 1 it. L.could g Bive as one of 


LEP would be unique. and therefore complemen- 
tary to other new accelerators planned-around. the 


atly 
under construction at the Stanford Linear Acceler- 


tory's linear accelerator, upgraded i in energy from 
the original 20 GeV to $0. Gev.."* to-collide elec- 
trans | and” positrons, Although technically ex- 
tremely_difficult (die priticipal problems is in. de- 
a eg se to ensure sufficient colli- 


ng the pete important 100 
GeV centre-of-mass energy region. The cost of the 


‘* If there was sufficient physics interest, the-energy of the 
SLC_could subsequently be increased to 70 GeV {this is 
referred to below as SLC-70). 


vst of the four additional detectors (pethaps anather 250) MSF) 


SLC was S estimated i in 1981 to be only $63 million 
(11, p.146}. Although revised to abou! 

in 1982 (cf. Trilling (40, p.52)) | still only 
about one quarter of that being spent on LEP to 


| authorization for the project to begin until 
al Year 1984 (cf. (19. p.81], but it had been 
funding the necessary R&D since 1980 (using the 
laboratory's operating budget). with actual con- 
siniction: beginning “in 1981 (cf. (15, p.8]). li was 
estimated in 1983 that the first collisions will be 
obtained by early 1987 (cl. Wojcicki (44, p.25)), 
just over a year ahead of LEP. 


3.4.3. $0 GeV CESR-I1 (Cornell, USA) - 


with the CESR -electron-positron: collider (see 
Paper -!1-(24}), Cornell scientists _began to. press 
their_case to enter the race to. discover the Z, 
Particle. proposing the construction of a second- 
generation electron-positron storage-ring (CESR- 
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a _ beam-energy: of! 50 GeV _might_ be reached in 
carly 1988, some months ahead of LEP, and for 
just over half the cost ($340 million - cf. Trilling 
[40.. p.40}). However. the. scientific arguments 
advanced forCESR-II_were virtually identical to 


those for the Stanford Linear Collider; and, in the 
Opinion of. the. large. majority of scientists: inter- 


viewed in the course o| of our study, the latter had at 


a third the price of CESR-II; it provided a test-bed 
for the new sacks of colliding linear accelera- 
develop 


unproven: technology _ oer cavities) 
that may not yet be suitable for mass production. ” 
In_view of this last point; Cornel] made it clear in 
1982 that they would not submit a firm proposal 
until they. chad. -Satisfactoril ily demons pons: trated . ate the 


ihai stage, Cornell capa to he i iy a 2x ite eG 
1984 to submit a. proporsi fo vorcnvncing: con- 
struction -of CESR-H-in Fiscal ¥ 


i ticular, a improve its luminosity) and 
to continue R&D on superconducting cavities for 
possible use in futare machines. 


z 4a 150-500 Gev VLEPP (Novosibirsk, USSR) 

The Soviet high-energy physics centre at 

Novosibirsk in Siberia has, as. we have noted 

elsewhere (Irvine and Martin [25)), Dc developed santo 
of excellence for 


from two fiscer paxolesgens Ata later stage. the 
length of the linear accelerators could be increased 
to give beam-energies of up to 500. GeV (ef. [17. 
pp.417~18))._ However. in view. ‘of the Soviet Un: 
UNK: it: would-seen doubtful whether substa: tial 
funding for VLEPP could be made available until 
the late 1980s. ?" Consequently, even if the project 
is eventually authorized, it is unlikely to be com- 
pleted before the mid-1990s. 
3.45. 1000 GeV Large Linear Collider (SLAG, 
USA}. : 

- Although: the. Stanford. Linear -Collider ee 


SLAC have already begun to consider a scheme to 

extend current _linear-accelerator technology to the 

1000 GeV (1 TeV) energy range. No definite pro- 

posal has yet been prepared, but such a project 
Subpanel 


billion (cf.- Wojccki [44, p37). Gi 
current commitment to the early ¢ completion ¢ of the 
SLC and the fact that the necessary preproposal 
R&D will take at least five: years, such a facility 
could probably not be completed until the latter 


part of the 1990s. 


4. A tramework tor Comparing the tuture prospects 
ica age eaaggg 


currently ee or undé construction around 
the world, ‘Weare now in a position to begin 


requency 
of LEP, but decided against this b"case of the attendant 
technological risks: “Superconducu, svities are not yet 
sufficiently mastered. LEP Phase I muss sscorporate ‘con- 
ventional’ technology ...“ [16, p.63). 


high-energy, 
perhaps been rcin‘orced 4 byt the evidence presented 
in the previous section concerning not only the 


2" Sitictly. the Novoubirsk Snatiiute belongs to the USSR 
Academy of Sciences (Siberian Branch), while Serpukhov is 
an Institute of the State Committee for Atomic Energy. so 


VLEPP_would be funded. from. a differeni-source than 
UNK._ Even sn. it seems unlikely that the Soviet Union 
could afford to commit itself to providing the resources for 
two very large high-energy physics projects simultaneoutly, 
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very substantial costs involved (and~indead-the 
possibility of failure as in the case of ISABELLE). 
but also the somewhat ad hoc way in which such 
facilities have tended 10. be - planned. authorized 


of some concern given the trend towards increas- 
ingly powerful institutional ‘Pressure groups dis- 
Cussed at the start of the paper. 

In what follows. we a a set of ihiricen 


tors -determining - the relative 


re of high-energy physics acceler: 


Sui 
ators am ( 
future scientific potential: but also take 


account factors uch as rélative resou Freq 


Over the next decade or so against these criteria. In 
addition, the recently completed proton-antipro- 
ton collider at CERN anc i i 
ISABELLE and CESR-II projects have been 
included for comparative purposes. Each machine 
is assessed in terms:of the various criteria on the 
basis of material and data discussed in the text 
below. As: will: be seen, this table ‘provides a con- 
venient_ means of summarizing the pattems of 
comparative advantage (indicated by + signs) and 
weakness (~ signs) that exist among the various 
Projects, patterns that.can then be used in arriving 
at an overall assessment of their future | Prospects. 


relating to the relative construction requirements 
of these various accelerators. 


4.1. Comparison of construction requirements 


4.1.1. Financial eriier 


Relative costs. In_assessiziy the merits of different 
Projects, relative cost is « early one of the principal 
criteria. The estimated qisis cin inilions «f Swiss 
francs or MSF) of the mits. aczsierarars and col. 
liders under consideratio:. wire piven in tables 
1-4 Of . the cheape.: cvs the CERN 
Proton-antiproton collider (200 %SF). the Sten- 
ford Linear Collider (240 MSF), ana ti2 Ferrite 
1 Te" accelera : Je Sheir tow 
costs reflecting the fact that they all make argni()- 


cant use of already existing facilities. In constrasi. 


the 50 GeV first phase of LEP (910 MSF plu 
further 250 to 280 MSF for four detectors) and 
ISABELLE ( (estimated to cost a total of some 1300 


Y is figure 
had ceased escalating) are both considerably more 
expensive. However, even their costs are dwarfed 
by that of the proposed US Superconducting Su- 
per Synchrotron and its associated detectors (some 
6000-8000 MSF). These relative levels of cost are 
surmarized in the first row of table 5. 


se ” sf resources. ore of the major areas of 
lure prospects | for an 


of “completion | - a factor -which is- often 
crucial in Structuring success and failure.In the 
case_of the first phase of TRISTAN, UNK. and 
HERA, the respective national governments have 
already authorized ihe necessary expenditure. For 
the Fermilab 1 TeV accelerator and collider. the 
‘Stanford Linear Collider, and the first: p! 
LEP. » the resources have been-promised, but for 
all these projects _ there. is Fate leeway for cost 


sibility to financial resources = pac in 


the second row of table 5. 


As accelerator con- 
ingly become a sophisticated 
logy activity, so the technical difficutty 
of constructing and operating such a facility har 
become 2 more important factor in determining its 
likely future success. “This. is_particularly true for 
very. adv banees technologies like ; Superconductivity. 


4.12, Technical criteria 


design and _Sonsirection of the Superconducting 
wired for sn. 1 TeV accelerator. Even 


then, the task proved far from simple 
only in 1980, after 3 lor.g | development. process, 
that they were able to begin ws snufacturing mag- 


2 Qace this fest phase ef LEP bits beda completed. CERN 
witl om all tkehhood be able 10 convince the Member States 
that, _father han: = Oocreastng the al budget to the 

? de used to increase 

ra ne twee HY aad ber 16 130 GeV: 
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TableS = : 
Companion on of major scceleratoe projeciié: Summary of man relauve comparat:ve advantages (+ + )and disadvantages (= = —_— 


CERN Fermilab Fermilab KEK - 


pp 


1Tev pp TRISTA: 


energy (pe mass energy ++ 
region Period of world leadership ++ 
Other _ Event ra vate = 
7 - factors Number of expenmental 
Scientific Boverning | areas ~~ 
potential ability ____ Variety of experiments 


lOgenerate | possible 


usersto | Scientific track record of 


++ + 
++ +++ a 
++ -- - 
a 7 = 
+4 = = 7 


eaploit msaiiae. = + 
Sani { sensain aos Beng 
pe Flexibility /porettiial ion fone 
_ development of scoderator selec 


ear signifies no particular comparative advantage or isadvaniege 


where the ISABELLE project required magnets 
with significaniy higher fi field-strengths, the prob- 
len 


tion programme, ?? and hence ultimately-for- the 
project's canceHation. The technical. difficulties 
faced by the CERN accelerator builders in at- 
tempting to achieve a luminosity of 10° cm~? s~! 


ba wr -noted earlier. one-of the factors contributing (0 this 


delay was the fact thai the Original design for the ISABELLE 


of ‘unsatisfactory development and testing. Part of the prob- 
fem-may. have stemmed from the Brookhaven design being 
80 different [rom that adopted at other high-energy physics 
centres that the laboratory was unable io profit izem the 


results of work elsewhere (cf. (11, p.146)). 
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completed at CERN ahead of thit US machine. 


ime proton-antiproton collider have also ‘been 
extreme, the machine initially: operating with a 
peak luminosity a factor of 200 below the des 
figure, although this was rapidly improved by < 
and a alt orders Of magnitude over the 


will enjoy the considerable advantage | of being 
able to benefit fromthe experiences at CERN, 
making their task rest ily simpler. In the same 
wav. Serpu aborative work with 
Sactay | on- the acaisiadasing magnets for UNK 
will te_able- to profit from Fermilab experience 
the magnets for the 1 TeV fixed-target accel. 


erator. as indeed will the builders of HERA: *4 


* The proposed magnet designs for doth. UNK and HERA 
ase reporiediy similar > that adopted at Fermilab (cf. (11. 
.147] anid J13; p:206)): 
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Table 5 (cond.) -- 


CERN Brockhncn. Serpukho. DESY KEK CERN. --- US 
= LEP.S0 ISABELLE* UNK HERA ep " TER.ON/130 SSC 
a 


Estima-ed date af firt cats ests a ee eae or 
“expenments 1988 = 1987/87 1990? 1960" 19607 1990;7 199057 
Cheapness  - -- -- - in ; = PT 
[Fnenca foe Of resources + ~<2 + ++ * 
Copstrucnon {Technology xequired relatively == 
requiremenis | — Technical srenple/uzdemanding + -- ry = 
ical Techincal irack-record of 
Uaboratory ++ -- -- os . 
—— $e 
4 Relative i increase in cenire-of- 
MASSENETEY = == + +++ = 4+ +++” 
{Penod of world leadership + -- ++ ++ -- +4 ++ 
Event rate. + ++ pal ra 
Sa Number of experimental 
Scientific Pare % * 
Potential 7 7 7 
= > ~+ + + - > 
++ - + + ~+ - 
Scientific track record of 
; ____}laboratory users + ++ -- : Pe + . 
ors}, 
Future Potential spin-off to 
Fowre fo GEE 
potential accelerator physics + ‘és S 
Flexibility /potenual f for future 


technical- problems associated -with the SLC are 
el icists to be particularly severe, 
d-it-will be a major engineering feat to achieve a 
sminosity of 6 x 107° cm7* s~! with the new and 
untested technique of the linear collider In con- 
trast. the task. facing the builders of TRISTAN 
and LEP seeinis more simple. For the latter, one of 
the main problems —- that of producing low-field 
magnets relatively cheaply - has already been 
largely solved (cl. [8. pp. 5-9). and the difficulties 


inherent problems in reaching the design luminos. 


35 Jn 1981, it was reporied that “CESR. PEP and PETRA are 
all having great difficulty climbing anywhere near their 


Rained. over a 1én-yéar period from smaller elec- 


design luminosity” 111. p.144} One. commentator co concluded 
somewhat ally that this “low luminosify siso cast 
a-shadow-on the performance ; f larger colliding 
beam machines still on the drawing board” (Robinson (33, 
p.1488)). There are perhaps two underlying causes io the 
results s0 far. One is thar calculations of the 
design luminosity. for the newly completed calliders were 


machines, and these extrapolations have subsequenily 
proved to be in (It is therefore: some: ‘alarmi 


Gune-shift value in calculating the eapected ixmunosisy” [33, 
p.1490] rather than the values actually obtained at PETRA 
and- PEP.) The other factor is that, in the rush to produce 


high-energy physics results as quickly as possible. those 
responsible for the operation and development of 
¢lectron=positron colliders have given too little attention to 
the machine physics involved (cf. (1t, p-144)). 
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associated with the vanous accelerator cia: are 
again reflected in the relevant row of table 5. 


‘Technical track-record of the laboratory. Oné of the 
best indicators of the likelihood of overcoming the 
technical . diffculties associated with esi g 
building. and o cali 
ous record of 


by- ie pivaieles we i interviewed in both Europe 


and the USA. In view of this, it was perhaps not 
unreasonable to assume when CERN embarked 
upon the construction of the proton-antiproton 
collider that_the very considerable technical prob- 


nee: SLAC's reco 1 -sug- 


4 sbably be able to solve | the 
problems -associated with the SLC and complete 
the facility by 1987 as planned. In recent years. 
DESY has had almost as strong a technical track- 
record as SLAC. Jt has grown into a major intéer- 
National laboratory. and with this has come. the 


breadth of technical and engineering expertise that 
will probably ensure it is able to. cope successfully 
with. the demands of HERA even though this 


lider appear beyond its technical capabilities: oe 

One ‘cannot be quite_so sanguine about the 
prospects for UNK.- As we have ‘discussed 
elsewhere (Irvine and Martin (25]), Serpukhov does 
not have a particularly strong record for building 
front-line research Yacilit robably for reasons 


to do with the structure of East European 


lems would eventually : solved, and ead 


of the-first experiments being postponed. 


science than with the ability of individual accelera- 
Althoug 


viding the sophisticated detectors and: power 


computing facilities needed to progress beyond the 


_, SLAC jie poseect 20 an excellent rack-record 


ade eer geen finan- 


ints (funds had to be found from within 
the laboratory's existing budget after the project 
application had: been turned. down. by the US 
Department of Energy). The more: recent experi- 
ence with PEP, however, was_markedly less suc- 
cessful. principally because of delays caused by 
outside See eapeccne contractors, a problem 


3 See the qt ation in section’8 of Paper II (24]-on the ISR 
being “widely regarded as the most perfect example to date 
{1979}-of the accelerator builder's.art.”-  ——— 
In_Paper_Il_ 124]. it was seen how the SPS's “technical 
superiority over the Fermilab accelerator was one of the 
main factors explaining the differences between the scien- 
tific performance of the two machines in the late 1970s and 
early 1980s. 


% 
% 


simplest “first. generation” experiments. Perhaps 
because of the limited access of East European 
high-energy physicists to the services of high-tech- 
nology. industry, it has been decided that_ the 


team, even with the aid of their French collabora- 
tors from Saclay. | 

__ Similar reservations applied to Brookhaven with 
respect to the ISABELLE Project while it was still 
Proceeding. Although Brookhaven had a good re- 
cord with the AGS two decades ago. the AGS 


quickly and efficiently overcome by” 
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Conversion Progranm- mdenakes ten years-later 


experienced severe problems (see Paper II [24)), 


and the record of-the laboratory with developing 


superconducung magnets -between 1976 and: 1980 
can only be described as poor. As has been pointed 
out. when the proposal for ISABELLE was being 
formulated, the machine had no obvious 

Competitor in-its particular regime of physics. and [the 


Project] had ume for RED - more ume_due to stringent 


happy with (Mew (31. p.18a). S 
gions therefore be considerable doubts about 


in-continuing a commitment to an eaacaois 
magnet design for several years after its limitations 
became apparent toathose inside the laboratory. 
Although changes in “the senior managemeni at 
Brookhaven ‘during 1981 and 1982 led to-some 
technological ‘revitalization of the project, they 
did fot completely remove what was probably the 
Foot cause of Brookhaven's relatively poor techni- 
cal record during the 1970s.- the failure to recruit 


capable young staff, especially accelerator physi- 
cists and engineers (cf. (6, p.247}) - which was in 
turn brought about by the absence over a longer 
period of any major new accelerator Project. 


4.2. Comparison of scientific potential 


: The scientific c justifications made for all the new 


similar, and exhibit a degree. of. convergence of 
theory and experimental practice seldom previ- 
ously witnessed_in the subject. Analysis. of the 
proposals for each of the accelerators listed in 
table 5 (and_ - of f subsequent. review | papers and 


facility) reveals a conimon emphasis on ter study 


79 In the sttitude survey described in Paper 11 (24), 74 percent 
of those cet earn with the statement. “It would 

if Bro had firs: spent a few years 

i f- superconducting magnets were techno- 
wgically (easible before investing considerable resources in 
the construction of ISABELLE™. weil over three times the 


— umber who disagreed (22 percent). a ae oe 

2° Prototype. magnets of the necessary field-strengih were 
successfully developed in 1982. although ai. the time the 
Project was cancelled it still had to be demonsirated that 
these were suitable for mass-production (cf. Wojcicki (44, 
p30). 


Of theintermediate vector bosons; the search for 
the Higgs particle and for new leptons and quarks; 
investigation of the quark-gluon picture and the 
theory of quantum chromodynamics. the explora- 
tion of possible “grand unified theories”, and the 
discovery of of unexpected Phenomenz. How, then, 


scientific potentials of these various accelerators? 
What criteria can be used? 
__ The history of experimental) ligh-ewergy physics 


main factors determining the scientific potential of 


an_-secelerator_are whether if opens up a_new, 


significant advantage i in yerms of increased cenue- 
of-mass energy over other machines), and the 
length of ‘time it enjoys a position of world leader- 
ship for that particular. type of accelerator. As was 
£een in Paper I] [24], this was certainly a crucial 
component in the success of proton machines such 
as the Berkeley Bevairon. the Brookhaven AGS 
(with its small but si cat 


Stanford linear erly aa among _electron- 


Positron colliders with the success of first SPEAR 


__A second set of factors, and one which has 
become more apparent in recent years, relates to 
the varying potentials of different types of acceler- 
ator for. generating new physics results. Several 
factors are of relevance here: the event rate (which 
Getermines the amount of experimental data that 
can be collected in a given period), the number of 
experimental areas or interaction | fegions (which 
determines the number of- experiments that can be 
undertaken 3 at any one time), the variety of experi- 
ments $ possible o on the accelerator, and the degree 
; the experimenta) data produced 
(ie. the ease with which they can be. interpreted), 

“Last, and of great importance, is the ability of 
the yser-group associated with the accelerator 16 
exploit its. full potential. As we have argued 
elsewhere (¢.g- Martin and Irvine (28}), one of the 
few indicators of that ability is theit recent track- 
record, and therefore this cntérion has also been 
included in table 5. 

Let us examine in turn cach of these criteria 
telating to the scientific potential of the vario 5 
accelerators under consideration. 


~ 269 


o.. 
ERIC 


332 


first the proton machines: one can see from table 2 
thai the centre-ol-mass energy achievable with the 


CERN proton-antiproion collider is 540. GeV. 


er. the CERN ISR_- an unusually large 
caressed by. any | other accelerator (the ISR came 
closest with » five-fold increase in centre-of-mass 
energy over Serpukhov). As for the rival Fermilab 
proton-aaliproton collider. its energy advantage 
over the CERN facility will _carrespond to a factor 
Of just over_three and-a half (2000 GeV compared 


with. 540. Bev). again quite large in- terms of 


synchrotrons. while UNK will in-tumn eventually 


possess an advantage of about 75 percent over the 


Fermilab accelerator. Finally it is worth comment- 
ing that ISABELLE, if completed. would have had 
a centre-of-mass energy very much lower than the 
collider at Fermilab, although (as is noted below) 
it would. have had the aslincica of a ee 


event-ra.e. - 


STAN's energy will be only 30 percent cise 
than that “of th the ay PETRA Mee table 4 


za co which, as we have seen, could be 
because it will permit detailed-studies to-be carried 
out of the mass, width, and | decay-modes of the Zo 
particle. If the SLC is completed-in 1987 as 
planned, then LEP will not have any energy ad- 
vantage when it begins operating a year later, 
though it will have othet relative strengths (see 
below). “However. if ihe_beam-cnergy of CEP i is 


achieved in electran—proc- collisions - 315 GeV 


compared with less than . GeV at the Stanford 


tiproton Collider. it is bighiy ranked in terms of 
this Criterion in table 5. 


pani Stage, these are-very approximate estimates 


only.) The relevant row in table 5 shews which 
facilities are likely 10 enjoy a relatively. long, period 


rating g reflects the fact that; for example, the CERN 
proton-antiproton collider is lixély 10 enjoy a lead 
of. some fi ive years over i its | Fermilab rival. while i it 


taken by.a yet higher-energy proton collider (per 
haps one pir ut pereee the electron- positron 
ring in the Li? to: oz the SSC). The Fermilab 
1 TeV accelera, >”, SUNK, HERA, a and the final 


ns due to the fact that they should 
enjoy a world lead of: about five years (+two 
years). ‘This is considerably more than the TRI- 
STAN and SLC machines, both of which are likely 
to be ‘overtaken by more powerful 
eleciron-positron_colliders_ within a year or $0. 
Finally. it. should. be_ stressed. that _ the: ill-fated 
ISABELLE project again ranked relatively low in 
terms of this criterion because it would have come 
into operation with a marked energy disadvantage 
relative to o1 chines as a result of the delays 


to the construction programme discussed earlier. 


2.2. Other Jociors governing ability of accelerator 
to generate new ald results 


Eveniraie = Te pradings ot the various accelera- 


tes, followed by proton-proton colliders 


(the proposed SSC is: likely to neve: a design 
luminosity of approximately.10°? cm“? s~') and 
electron-pox:tron storage-rings like TRISTAN and 


SEP (with design luminosities in the region of 
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30" = 10" m7? 2s po The event-rates or ihe 


Out On these machines iake Lead ane 
and consequently there are fewer of them. Con- 
versely. pi can be- used to make 
relnvely quick scans of large energy regions. 


Number of experimental areas. Fixed-target accel- 
erators like the Fermilab 1 TeV and UNK alio 
possess a marked sdvantage over other sorts of 
experimental facility in that several beams can be 
obtained simuluseously. As a result, large num- 
hers of experiments can be run concurrently - far 
the 
number: of interaction areas is strictly limited dn 
the final phase of LEP, for r example. eigkt experi- 
mental. areas are planned, while-in the first phase 
(LEP-S0) ieee willbe only four. At the CERN 
tiproton colliders, the 
number of pe is even more constrained, 


with just two intersection regions, while at the 


ford Linear Collider there will Probably be 
only one collision point, at least vans 2 


portant advantage over colliders eecase ty can 
be. used to. produce a wide range of s 
beams ( 


1 the-case of the | SLC with-its relatively. ibe 
luminosity 32), This has a number_of important 
policy implications which should perhaps be briefly 
mentioned at this point, though they are treated iii 
more detail in the conclusions. The first is that, 
with electron—positron ‘colliders, there is a crucial 
advantage to be gained by being first into a new 


Sd ii ie case of LEP.90, ie donates 610% ca 
hae "Tae tamwnoaley bile antber plane {LEPS0) fos: 
peed tc 31 em” 7's~'| giving it a probsble_ad- 
in this sespect of about 5 over the similar energy 
.. Stanford Linear Collider (SLC-50). - ----- 
°) 1 is envisaged that this may subsequently be increased to 
-- two, 


ill, however, have the compensaung advantage 
Shat_it.it uniquely suitable for polarized beam experiments 
{ef (12, p.201}). 


ogi i Toad (of jun ous or 00. yaa 
Z machine can be more important than a 
Tead-st f three or four years for a _fixed-target 
accelerator where it takes considerably longer io 
exhaust the range of experimental possibilities. 
Hence, the fact that the SLC is due-to. reach ihe 
100 GeV centre-of-mass regic.. a full year OF so 
ahead of LEP could prove a major handicap dur- 
ing this first phase. of the CERN machine. A 
second policy implication has been described in 
the following terms in relation to LEP: 


There is 4 natural tendency when inve 
of physics to be the “firstest with the mosiest.” Since it is 
Possible to define all the charscierintics we would like to 
know. about the products of clectron-positron collisions, 
this tendency could lead to all experimental proposals being 
built around nearly identical “universal” detectors. [14, 
p.24t] 


However, CERN management seem oe 
that such a tendency, if nat checked, could Jeave 


change i in physics interest, sirice they have urged 
that. the differences between. the Various. detectors 
proposed for “LEP be made rather greater than 
first seemed likely (cf. Robinson [38, p.722}). Even 
50, the variety of experiments that can be handied 


: , possid’ 
new a accelerators planned to come into oper- 
ation in the next ten years. 


“Cleanness" of data, Where electron-positron: col- 
liders do score highly aver other types Of accelera- 
tor. is in terms_of the ease with which the resulting 
experimental d data can be his 


action 1 between 1 two apparently point-like objects. 
each with precisely | y EZ 


between electrons and positrons nvclve an intc:- 


lucing “pure” energy which is then available 
for the creation of new particles that have a known 


e in n respect of the - Stanford ‘col: 


lider. SPEAR (a year pre of DORIS at DESY), and later, 
when ihe positions were reversed,_with PETRA coming into 
Operation Iwo years ahead of PEP. The rapid obsolescence 
of electron and electron-positron machines is further dis- 
cussed 1 Martin ane tivine [27]. 


oe 
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series of different. values. particular energy re- 


then it skould be ¢ 
In contrast. the-c 


of three quarks-and a number of gluons. In the 


resulting multi-body interaction. only a fraction of 
the collision energy is available for the production 
of _new particles; and since the energies of the 
individual constituent quarks. are unknown, jit is 
impossible to.“ tune” the experiment to. investigate 
nergy state. Instead. a2 whole spec- 
of different.e energies 


mous: growth in recent: years of available comput- 
ing power, one can never be entirely sure that 
pathing has-been “missed” in a given energy re- 
gion, * Last: HERA: lies somewhere in between 
electron-positron collideis on. the: one hand, and 
proioa “machines on ihe o 

using electrons - i.¢. apparently point-like objects 
with reasonably well known Properties - asa 
probe to L vestigate the structure and properties of 


of ‘users to sexilat accelerator 


As noted earlier. the Atack-record of the user- 


te revsely because soe = -ptob cena “with 
data-handiing in protce ‘that critics of 
ISABELLE argued that it 3 + the only remaining 
advantage it nad over rival - .nnes - wa: only of marginal 
tmportance. 


Thus. particularly high credit must be given tothe 
User-communities associated with Brookhaven (the 
AGS) and Stanford . (the: lines accelerator and 
successful records in world lenis. - each. was 
responsible fora ge anigieail large i number .of 


currenti ntly engaged in a majrr new accelerator | Ppro- 
ject (see Irvine and Martin [25}). However, these 
data on ‘past performance cannot legitimately be 
used in prospective evaluation without a number 
of qualifications first. being stated. In. the casé of 
Brookhaven. for example. it should be noted that 
the main achievements of the: AGS were made i in 
the 1960s by a.group of Predominantly 
Coast experimenters, while the user~ 
for ISABELLE (if it had finally been co 
1987.< -or 1988) would have t 


gh 
heyday of SPEAR. most fats its is principal Gace were 
employed either at the laboratory itself or at the 
Lawrence. Berkeley Laboratory. Since ther. the 
Stanford ‘user-community has been considerably 
broadened; *” a trend that seems likely. to: con- 
tinue, In_ general. the. future will probably witness 
even Bremler CTOSS-UuSAge of accelerators than in the 


facilities means that there will be less duplication 
of accelerators between continents. So, for exam- 


27 This was a straiegic decision taken in the latter. part of the 
1970s by the SLAC management, who became increasingly 
concerned-that; in a period of limited funds, the scientific 


case. tor. the laboratory’ . future ‘support might t be swamped 


experimentalists whose “research aclivnies were con- 
centrated around the more extensive fixed- -targel pro- 
grammes of ihe two national laboratories at Brookhaven 


and Fermilab. 
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os i numbers of. European: “physicists (pet- 
haps 400 or so = see Robi (39. p.814)) are set 
to use the new Fermilab machines. especially the 
proton-antiproton collider. * once these come i into 


™* while- LEP and HERA Tak be drawing 


Epis: experimentalists (which may, Bive rise to 
an-clement of competition between them). Such 
migrations of physicists between centres are, there- 
fore, likely to go at least some way. towards 
smoothing out _the differences. between the ability 
of traditional user-groups to exploit their experi- 
mental facilities, although the users of LEP. will 
sull 1 be agora West European ( (perhaps 75 


ortant factor in predicting future research perfor- 
mance. 


4 3. Comparison of foswre potenial 


ure | prospects of a major 
tor project. These concern, first, the likely. spin-off 
from a new machine to accelerator physics more 
generally - for example, whether a radically 
; ive technique for co’ - is being 
pi May usher in a new generation of 
research facilities; and, second. the “flexibility” of 
the machi - ‘tha! 1S. jtS: potential for future 
development (such as udding a new ring to gener- 
ate further _ experimental . Dassibilities) “or for 


the physicists . interviewed particularly stressed 
these factors. arguing that. if a Particular new 
“ergy-region should prove barren, ii is mmportant 


”* There has bes some. discussion, for cxample. about the 
possibility of moving the very large CAI detector from the 
CERN collider to the larger US facility when i nears 

— completion in- 1985. 

™ Of the four_LEP collaboraiions Bven the provisional go- 
ahead in 1982 by CERN. one ts largely American (cf. 


Robinson (38. p.722}). 
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to be able to wake alternative use of what are 
highly expensive capital facilities. 


Potential spin-off to accelerator phi 's°. Of the:vari- 
ous accelerator projects considered uere; the Stani- 
ford Linear Collider probably. bas. the Breatest 
potential in terms of spin-off, since it iz 
opportunity. for sesting. the feasi sibil 


ogy is important because it now seems. likely that 
LEP willbe the. last of the arg 


jarger ly expensive 
to construct ary cost--¢ oF such machines. rises ap- 
proximately with the square of. its $ energy). ~but the 


Overcome the: energy: losses through ‘synchrotron 


radiation). Furthermore, the heams would be sub- 


at centse-of-mass energies greater than 350 GeV or 
$0..the most promising way forward would a appear 
to lie with colliding linear accelerators (the cost of 
which rises only eal with energy). According 
to-many of the 


proton—antiproton: collider has been _technologi- 
cally-very innovative in that it has involved devel. 
oping the technique of “stochastic coolitig”. whi 
has the potential for extensive future applications 
The main. a. fromthe Fermilab 1 TeV a 


Superconducting 1 magnets. Finally, the last phase 


“ Indeed, it will “take over 230 MW at cleciniciiy to power 
LEP at-90 GeV. the equi 1 energy. consumption = 


medium-sized enty, At -was 


American accelerator-physicist._inumasiely th the 
development of the SLC, referred 10 LEP in an interview as 


! from high-f field 
gh one another at 


very high energses.__ 
“7 Ue és significant that the SLC i is categonzed asa research 


and development project,” a it will have “a timely 


physics payoff as well” (Wojcicki { 
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ng, ta dio-frequency-ca it es. as would 
CESR-I]_had it been built. However. in none of 
these cases is the magnitude of the potential: spin- 
off as great as for the Stanford Linear Collider: 


been-made sufficiently 


> developments, thus minimizing the 
engineering costs associated with any major 

new ‘project of this sor.) As we have already 
noted, by using superconducting. magnets, a pro- 
ton synchrotron with an energy of up to 10 TeV 
could be built on the CERN ‘site: This accelerator 
could if irr be converted into a large proton- 
antiproton collider and be_used. in conjunction 
with LEP. to collide 130 GeV electr 10 
TeV. protons. This potential for future.develop- 
was referred to. by many of the CERN 

98 


guarantee the long-term scientific future lor CERN 
(and West European high-energy physics) by virtue 
of the wide range of new accelerator developments 
that can be accommoda‘ed at the Geneva. site. 
However. it is significant_that, since these inter- 
views were. conducted, US A sag =A physicists 


Physics Adio Pana uianudl endfead 
ieroamne:dstion of their peabpanet on New Facili- 


wih <aiabliihad dachaology -with the result thatthe spin-off 
10 accelerator physics is unlikely lo be particulary | signifi 
- cant, - os 
“As Wilson (43.-p.37} has-observed, “this potentiality has 
nol _yet been publicly mentioned by the proponents of 
[LEP[”. a situation that only began to change in 1984,” 


“4 


achievable with superconducting magnets in the 1990s. 10 
tesla magnets would be sequined 10 acceleraie — lean 


p.26)). 


rojet s 
priority” if the. US programme in order to achieve 
“completion in. the “first half of the 1990s” 
(Wojcicki (44, p.4]). If this tangs: 1s met, the SSC 


n“nts-in favour of proceeding: with the construc: 
tion of such a CERN accelerator would-be ‘very 
substantially reduced. In short. the “real-estate” 


force if construction of the SSC proceeds . “apace. 
__ As was shown earlier in. the right-hand column 
of table 1, the ‘new proton synchrotrons at Fermi- 


as with LEP. With the proton-antiprotot 
at CERN and. Fermilab, however, the -pos 0 


and disadvantages | of the various new facilities 
have been summarized in table 5. 


5. Overall assessment of che fone prospects for 
CERN and its users 


__ Having evaluated i in i detail the prospects for the 
I fac 


CERN and indeed for Wer = op hieh-anerty 


dnved ap collider. To take the first of these; the 
SPS: user-community- has, since the late 19705, 


menters at Fermilab; these include amore: reliable 
accelerator, higher-quality beams, meee more 


it shld be noted. however. that the development of th 
CERN collider has paved. the way for the- low-energy 
i ility. In addition. some thought 


expense of. substantially reducing the available luminosity. 
and, at we have seen, this is still rather low for ceriain iypes 
of important experiments. 
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sophisticated detectors. and a higher level of finan- 
cial support (including k Operating - at 
Fermilab ‘this was drastically curtailed during the 
early 19805 in order to limit the izboratory’s elec- 
iricity Cosis). However. nov that the new Fermilab 
fixed-target machine has started to reach energies 
in excess of 500 GeV 

‘Ils Yesse da 


ative scientific out- 
accordingly.- : 


and disadvantages are. well summarized 
in table 5. When CERN-too¥ “the deci t 
embark on this project.. the. technica’ 
seemed formidable. and indeed there w 

ment of risk ihat ihe design wo 
prove unattainable. Balancing 1 jowever, was 
the fact that CERN had in the past two decades 
developed probably the greatest body of accelera- 
tor expertise among all the major world labe. :10- 
Fies. As the project manager for one of the main 
US accelerator programmes remarked, 


On balance. 1 suppose our acceleraior construction team 
fanks with CERN's firs. team. but the}_have iccond. third 
and fourth teams as well ... no-one can match that range of 


expertise. [Interview, 1981] 


song new accelera 
tors. there were good grounds for optimism when 


ider's relatively Ic 
ple be expected to restri 
it-can ca 


1e Tange of physics that 
jump in energy it repre- 
its comparatively large lead over the 
r_at Fermilab, should ensure that a 
fairly wide range of experiments is Completed dur- 
ing the period up-to.1986: Moreover, the CERN 
collider has proved relatively cheap in terms of the 
investment How fequired for a iypical.major new 
experimental facility, even if it et at 
the expense of 1 redu rof 
the SPS and the time able for the fixed-target 
Programme. This combination of relatively large 
scientific potential and low cost would suggest 
that, even though a high degree of technological 
uncertainty was involved, CERN chose wisely in 


1978 whea embar: ng 


y ) 
Of the physicists whom we inte: viewed in 198) ane 
1982: Pe te 
~ In contrast. the comparative advantages -and 
disadvantages uf LEP are almost the exact oppo- 
Site of these associated-with the proton-antiproton 
collider, as is apparent from the pattern. of 


strengths and weaknesses summarized in table 5: 
First, the technology required is relatively con- 
secvative and simple, apart from thé. tunnelling 
where civil engineering problems. might possibly 
ive-rise to some delays. Oni the other hand. LEP 


Many limes more than 
ber States have agreed that 


first 
tecommended that LEP should be CERN's next 
Major machine. In ‘particular, the Intersecting 
Storage Rings were closed ai ihe end of 1983 - a 
Major ‘sacrifice given that the cancellation of 
ISABELLE: meant this would have 
unique accelerator unti! the 


pre- 
see table 6 in particular) 
__sh worthwhile experimental work 
may still hase remained to be done on the ISR: 

_Third, and last, whereas the Proton-antiproton 
collider represents a significtiit increase in cenire- 
of-mass energy, and. one wizch will present. its 
users with @ position of + poly in the energy 
P vea8, the first phase of 
LEP will achieve an energy ne greaier than: that 


likely to be attained just over a year earlier by the 


“7 There would wave been a ‘+g case for eventually devel- 
oping the ISR as a heavy wun collider. which would have 
Produced another unique facility for a relatively small 
investment. 
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produce over a million Ze events per year: in other 
words. the SLC shouid be able to carry out detailed 
studies (with reasonable statistics) of the produc- 
ion and decay-snodes of this crucial particle. (cf. 
[12. p.200}). This latter point has not always been 
recognized or admitted by the keenesi supporters 
of LEP. as this CERN physicist’s comments make 
clear: 

The local “party line’ a1 CERN is to doubt the SLC. They 
hope it won't_work at all: But if it works wothin a factor of 
10 oF even 20 of the promised luminosity. and works in 
1987, st will get a few thousand Z, decays. This will be a big 
psychological burden for LEP. Some. people on the LEP 
Project won't face upto inis.. The story. then follows typi- 

cally that. at least LEP will be able 1¢ do precise measure- 

ments. Then people start talking about Phase 23. Bus that 
of money. and_will have only a very low event 
ratc ... So the usual CERN view is to say that the SLC 
won't work. Also one of the views ca the LEP tunnel. 
reconci is the ‘real estate’ view — the possibiiny of 
Putting @ proton accelerator and a _Proton-antiproton.. col 

lider down there. This Softens the view 

SLC as wel 


"d 
Stop-_the crazy international cumpettion. t-has beer, 
for SLAC. losing the race with PETF_. It LEP is loser jin 
the competition with the SLC}. this will be terrible for 
Europe. CERN: muss win the competition. But rm pessimis- 
tic - the SLAC machine will probably just spoil LEP. 
[Inte :ew, 1981] 


solescence. Ceomaay. _a lead of just one_year 
for the SLC could prove extremely serious for 
CERN in that the research programme for LEP 
could consist very largely of repeating. SLC experi- 
ments is a more detailed form: In other. words, 
LEP might be tesiricted to the same Sort of role as 
the CERN. SPS performed in relation to. the 
oe 400 GeV accelerator. While such tasks as 


n actually making a new dis- 
cov ng-out the first measur 
Particui.:: particle property. Thus, it is difficult for 
the outside observer to avoid the conclusion that 

vientific potential of the first phase of LEP is 


: j on .ct eves a luminosity 
close to the design figure relatively quickly. In that 
additional | justification for the consid- 


completed on .chedule and 


be pert 2!sewhere - for example. in later. pnases 
of LEP and in possible prt pla 


above on the effect on th 
SSC). is 

_oIn view of ihese iather uncertain prospects ig 
LEP. it is worth asking why enthusiasm for- the 
project apparently remairs so nigh. One-possible 
explanation is that, jus: as the ISABELLE Project 


relic of high-energy physics interests in the early 
1970s “ (before the technology was available to 
construct. large Droton-antiproton colliders). so 


thusiasm for circular electron- =positron colliders 
was at its height. Then. a large storage-riig like 
LEP appeared to offer. the best. prospects for, if 
not discovering the Z, predicted by the newly 
thumphant ‘Weinberg-Salam theory. at least in 
vestigating its Propertics | in the | “cleane 


early enthusiasm for ‘LEP Still persists in, subite 
although ‘some physicists have begun to. harbour 


rivate. 
about whether the scientific case for LEP: Phiase | 
is still as strong as it was, say, in 1978: These 
doubts may at some stag surface ¢ in public if the 


the level of resources a 

other experimental programmes. 
Before we offer concluding remarks about the 

relevance of the present. evaluation exercise for 


ni - 
servations abou: e future { for CERN are perhaps 
in order Of special note is the fact that. apart 


from ‘ermilab 1 TeV machine. all the main 


oT semdence in table $ to suggest that the United 
wrong 10-halt the construction of ISABELLE in 
200 million alzeady spent. 
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s are colliders. This has pro- 
ions foz_the extent and siructuré of 
experimsr - _atgh-energy physics activity. since 
only a ve., limited number of experiments are 
possible at any one time on a collider. As a result. 
the present tendency for the size of collaborations 


ment is sapparenily é Jet anyone ahs wants to use 
the accelerator join one of the first four accredited 
1 


Europe's 2000 high-energy physics researchers will 
be working | On just four experiments, each with a 
lead 


experiment to the production. of scientific 
The _same trend is likely to emerge in the 
United States. Particularly if it is decided that the 
country can no longer afford to support three 
national high-energy physics laboratories. 

The task of managing these vasi collaborations 
will not be ‘without. its organizational and «x s0cio~ 


logical | problems: i icreasing 
ing s 


; in _ integrating 
te students into ‘an experiment ‘whose 


- stretches far beyond. their training 
d so on. Indeed, jt is the growing TECOg- 


an padi over the US centres since it has had 
mort experience in dealing. with such problems 
Over tne fast 25 years. Hence, CERN is likely to 
suffer Jess from tbe effects of these problems on 
scientific. performance. The advantage formerly 


ing political ‘demands associated with attempting 
lg serve thirteen Member States) may disappear. 
Wit y then go the success previously enjoyed 
by Ur Sta in making most of 


the crucial discoveries in experimental high-energy 


physics during the 1960s anc 1970s. 


6. s ee ra Sens + ~ding remarks 
As will already be clear, our overall assessment 
of the future for CERN i is not a wholly optimistic 


Pointed out. This we fully expect, since any oa 


sion of the future is inevitably clouded by uncer- 


ides: anal} ing the future prospects for 
CERN. we vould-also sope that the paper will 
succeed ii in focusirg attention on: what many now 
believe are crucial deficiencies in the decision-mak- 
ing process in Big Science. 4 recent US repori 


noted that 


there is some Sotting iba tae high energy physics commua- 
ily does not have .dequate opportunity to participare in this 
Planning process which so affecis its future. (Trilling (40. 
p39) 


However, potas of even greater concern is 
that those ouzside the specialty have even less say 
in decisions that can now iuvoive the commitment 
Of hundreds or even thousands of millions of 


cr lars. It is. perhaps somewhat. Surprising in an 
age where State. expenditures Awe Soon: under 


cations for other areas of 5 science, was given the 
go-ahead with apparenily 6 so little real discussion 
in f-- lic oF even wi'hin the scientific community 
of high-energy ph 
tive 
“extension” of existing facities, thereby curt ling 


debate about the merits of the nroject and indeed 


abou the future of the CERN: Iaboraiory as a 
i ely a somewhat debatable precedent 
from .he point of view of publicly accountable 
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aul the faites commie each aca eel 
that virtually. the only option remaining is comple- 
tion of the. Project. .__ 


those ateas of Big Scence characterized by large 
ci ities that structure the inter. 
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Member States (and not just by high-en= my physi- 
cists) before any major expenditure on LEP had 
been *acurred.__ 


7 a should perhaps be. stressed shat the aim of the 


a stronger competitor in the international “race” 
with laboratories in. the United States ana 
elsewhere. - - ~ the cosis.of the wastage involved in 


Yet under the-conditions of “oligarchy” encoun- 
tered in Big Science, it-may become increasingly 
difficult to locate researchers capable of providing 
=_and. more impoftantly. being seen to provide - 
the _ Objective and disinterested. judgements on 
which peer-review. mechanisms depend. = 

-It_is inthis concat that we have explored the 

= of external assessments of the sori described 


national coordination: of ' research efforts.” The latter 
is one of the concerns of the International Com- 
mittee on Future Acceicrators (ICFA), yet up till 
now Uns body has c-en largely thwarted in its 
efforts by the continued emphasis placec by physi- 
ig” the next international scienufic 


of wider. public and: scientific: :pafticipation into 
decisiun-making. Moreover, given that it is very 
much_a first attempt to tackle this task. it may be 
subject fo criticism. I is worth emphasizing. how- 
ever, that. the peep 4s not to replace the peer- 


publicly available, it might-have strengthened the 


case of-those the-.  sguing in favc vr of CERN 
pursuing the proton-antiproton op.on. Similarly, 
a_comparison of :uch a table for 1978 with one, 
Say. for 1981_ would have revealed to high-energy 
physicists and &< ky public alike tha< the likely 
scientific potential for LEP_had been appreciably 
altered by the proposal of the Stanford Linear 


gional collaborat'on. Most in 
TeV proton synchrotron, and various. technical 
studies cf such an accelerator were subsequently 
sponsored by ICFA. In 1983. however, US high- 
energy physicisis began to urge their. Government 
to build a proton machine of precisely this € 
in order to ensure that. their 5 previous “d 


sumably also i in _Eastemn 


few years time with plans f for an even, larger accel- 
erator in order to ensure that the “lead” in high- 


* See also the disr-sssion in Trilling (40. p.3] of the need for 
the United States 10 construct a substantial new facility 
over the next few years in order to maintain sls “ pre-emi- 
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— physics is iS not once agai losi to the United 


i+ bre this process of escalation (which has 
several seatures in common with the “arms race” 
between the su 


powers) will-end-is-by no means 
clear. Yet at some: stage.the governments of the 
three “super-powers™ in: high-energy physics must 
surely cal] a halt on the gfounds that the cost of 
the “next” accelerator (which will be measured in 
billions of dollars rather than millions) ig 100 great 
for a single region to bear. High-.- gy. physicists. 
who nave for m:+y years discu vague plans 
for a “world accr:trator™ only. for researchers in 


one. region . “to to build auch a mach government (or 


Bional) consierations (such as-attempting to wrest 
* the lead” from some other region) should play no 


part. * if this paper. succeeds i in stimulating discu- 


generally of . hs probler:.: associated with existing 
decision-making mechanisms in Big Science, it will 
have performed a useful function: 
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; ABSTRACT 


This j paper presents the results of a study comparing the past scientific 


performance af high-energy physics acceleratorsin the Eastern bloc with that 


- Of their main Western counterparts Output-evativation indicalars ere: used. 


After carahilly exainin:1g the extent to which the output indicators used may 


be biased against science in the Ea. tern bloc. various conclusiors are drawn 


about the relative contributions to science made by these accelerators. Where 


Significant differences in performance a are appare' ., a an ss fs made fo 


Basic Researchi in the East and West: oe 
A Comparison of the Scientific Performance 


of High-Energy Physies Accelerators 


John Irvine aid 
Ben R. Martin’ 


Pitas i in both the East and West rc ¢ increasingly being forced t To 
consider how best to formulate appropriate and effective policics for 


science and technology.” One responsc has been thc developincit of 
improved techniques for monitering rescarch outputs — whether 
they be-contributions to scientific Anowlcdgc, or to technological 
innovation and economic development: Over the past five years, we 
have undertaken various studies of the scientific outputs from major 


research facilities in several Big. Scicnce- specialtics (radio 
astronomy,. optical pias ang clectron. high- -cnergy. physics). 


bloc over tlie 19605 and 19708. While we give particular attention to 


Social Swaies. of Science (SAGE, London, Beverly Hills and New Delhi); 
Vol. 15 (1985), 293-341 
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the major accelerators at Dubna and Se Scepukhov, we c alse consider 


low-encrgy rescarch facilitics at four cthe: labosatere 4, eo that we 
cover a very large part. of the Eastern Hlas's oft "ifr ypcranental 
high-criergy pliysies. Our evaluation *) oc be Sie inethod of 
“converging. partial. indicators’, deceit: ila shed So iavertves 
cor dhe 
United States in terms of theiroutputs of research pi fades ins, the 
subsequent impact of those publications on tie. asvaner 2% scientific 
knowledge (as indicated by various citation sal Gerd, and 
ean Hl ache Al data obtained from tiie “iews with 


comparisons. with similar_facilities in Wesicr Dive 


perlorniancs of Eastern-bloc. science: Some. - nsajor Senet: 
identified as- sreairicting research i in n the Eastern bloc are * Hiscusscd, 


scicniists: and. iiiteraational lata. -banks on_ peels and 


citations. These limitations arc then taken into consideration when 
the differences in output between Eastern-bloc. and Western 


_ accelerators. are analyzed in detail. We argue that biases in thc 


varions indicators are.-in fact, insufficient to account for the 
apparent differences in the oulput of similar rescarch facilities in the 
East and West; this.conclusion is supported by our peer-cvalaation 
resulls. Finally, we attempt to identify some of the pritcipal factors 
determining the: scientific Eperformmcest Easte rit “bloc accelerators, 


energy physics coniniunily. 


The Gr owlh of Interest itt Fusters- Bloc Science 


Two motives appear tc diinderiic seh previous work on Soviet and 


Eastern-blac science. First isan obvious curiosity inthe dynamics of 


a social system which, since 1917, has pursued a novel path of 
development, based in large parton policies attempting to make 
‘rational’ usc of all resources, including science and ter} ‘nology.* The 
Soviet revolution had profound ramifications for scientific research, 
and there was an upsurge of Western intercst in. the Soviet rescarch 
and development system in the 1930s, following the congress on 
‘Science at the Crassroads’ held in London in 1931.$ Stalinist purges 
and the rise of such ‘distorted’ sciences as Lysenkoism provoked 
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people’.’ _ 
_However, infornicd. “understanding of Eaten big’ science really 
oniy t began with, first, the exodus of emigré scientists with first-hand 
experience of wort vig in research lahoratorics, and, sccond. the 
development (in tlic early_1960s) of exchange visits by scientists, of 
international collaborative projects," and of conferences attended 
by researchers from East and West. This growth in personal contact 
allowed scientists to reflect upon the merits and weaknesses of cach 
system, as well.as opening. up. new opportunities for those mudying 
comparative science policy’ and the sociology of science. _ 

. A_secotid. motive is reflected in the emergence of systematic 
monitoring by the West of Eastern-bloc science, since this is now 
regarded as a. key strategic elemcat in industrial and military ry policy. 
Both at a public level (through organizations such as OECD), and at 


more covert levels, significant efforts have been devoted to studying 


Strengths and weaknesses of the main. Eastern-bloc R&D 


, seclors,.and of the associated basic sciences. Such stiidies have not 


only considered the inputs to science and tech nology (funding levels, 


numbers of scientists and engincers,"" and so on), but have also 


altenipted to evaluate the outputs from R&D; particularly their 


potential value to. military technology.'' Political interest has, at 


times, clearly influenced the. provision of government funds for 


academic studies of Easterti-bloc science. 


What We Know about Fastern-Bloc 
Basic Research Performaasy: 


Despite the recent proliferation of bierauien cn Savic: and East 
European R&D, there is still remaz ably little ‘hard’ quantitative 


data on the performance of its scientificsystem — certainly titite that 


would be gencrally accepted in botit East and West. Pari of the 


problem is the reliance on personal recoliections cf erigré scicrtists 
which are, almost inevitably, strong'y influenced by politicat values. 


The_ impression gained from emigré writings is ane -of ch onic 


inefficiency, with the underlying cause located-in an all: pervasive 


bureaucratic structure, the operation of which discotirages ¢ crealive 
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and = productive scientific rescarcin, __ especially _ aniong 
expcrimentalists.“ The emphasis on tong-tcem planning, in 
particular, is widely criticized on the grounds that it is difficult, if not 
impossible. to match the ever-changing and often uapredictalle 
demands of experimental rescarch with a relatively inflexible fong- 
term. beac on. ae identifi cation of — 2 problems. 


sitibulahicl Te) areal dysfunctions: these ada those faced 
by creative young scicntists in challenging conventional wisdom; the 
generally poor level of scientific managenicnt (in turn relatcd tothe 
aver-centralized control of research); the stagnation of research 


institutes duc to ‘ageing’ of their staffand a lack of mobility between 


laboratorics; over-concern with secrecy, inadequete binks between 


the Eastern-bloc scientific system and the rest of the world," and, 
finally. poor links between research instites. = 6" high- technology 
industry resulting in the. suinied develop... 2° many areas of 
experimental science denicd modern instru’. 2“! “1 and adequate 
computing facilitics. In short, cmigrés are se *:Carcas of basic 


rescarch (with only a few exccptions) are t.cciapetitive with the 


West hecatise. the conditions essential for successful rescarch are 


generally lacking. - F 
a ea of view + af f comparative scietice= policy analysis, the 


eats wr ritings: alt hough visitors clear ly ave! Icss biniwledac about 


the detailed operation of thescientific system, they arc morc likcly to 


hold ii balanced | view, " * White visitors tend to confirm reports of 
ning facilities, a ines 


aneae ly. the theorctical branches) the penondaice ot Fastern- 
htoc: rescarchers has heen meet the hest.it in v tHe 9 wor aid, at least 


theory... 


_ Prominent among the initiatives aimed at proinotivgeolahorative 


Prcigraiii, which has been. reviewel dicgettics with other aspects of 
Ausciieae “Soviet scientific Ela aa eh the US - Natisinal 


caprcricnees: and io comment iipon ihe quality « of Soviet scientists 
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sss gach 


7 


scientific outputs, particularly. since the appearance in Science 
Indicatars 1972 of field-by-field. counts of the scientific papers 


published by the major industrialized countries. and figurcs on the 
frequencywith which they are cited.” According to those who 
produced these data, 
+. Publication counts “ere used asindicators of national scientific activity while 
the citation counts were used as indicators of national scientific ‘quality’ or 
“significance’.?! 
On thesc indicators, the USSR coniparcs relatively poorly with s#:-s: 
other major countries, and in particular the. US: Such findings zrc 
now integrated into the conventional wisdom of Western science 
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policy, as the following stat atement { reveals: 


ny. any ‘measure —) wh iether Nobel prizes, frequency of citation by fellow 
specia 


rigin of major breakthroughs, or simply quantity of publications — 


US scientists tead their Sovict colleagiics in inosi disciplines, and in iiany (here is 
2 


simply no competition. 


Uneritical accepuait ce of these figures as.indicators of national 
scientific ic (pelea is, he-wever, extrenely fan a since the 


eapcenally true for comparisons heiwecn East anid West: tlie 
coverage of journals in the SC7 is very unevenly distributed across 
countries, with Enel een nations ns are ual well: In 


ple 


Lending | division, 3 only 14 of the. Saiict jouriale (less i 10 
percent) compared with 40 for the US (over 20 percent) were 
scanned by ISI — a discrepancy also evident in abstract counts.’ 

Stich hias is likely to Iead to underestimates of Soviet publ: lion 
output. IF physics ts bles then Soviet jevirials are onder: 
sideconey of the 1S! data as_an indicator of relative iiite: F ational 
scientific activicy was forced to.conclude that.in ‘the case of the 
USSR, the coverage is. so- deficient that internation. <ctivity 
indicators based on this Soviet coverage. would be «riously 
affected’ 2° Biasis aise inevitably present in the citation ia. Mfless 
than 10 percent of all Su:*et (but_over 20% of US) physi::s journals 
are scanned by IST, then Soviet physics publications are ‘tesing’ a far 


larger-ntumber of their citations (in the unscanned jous.sais) thanare 


ea ones. ASI- based data purporting tor reveal tlic love. any 


ecplicees aSc aia on n relative. niimbers of publicatio s. 

The overall conclusion is that the evidence available on the 
a: Tayimaei > if Eastern-bloc science is not at present sifficiently 
icanhle to support the crilicai views - often heard in. the West. 
Publication. and citation data apparently: corroborate the views of 


emigré scientists, the impressions gained hy Western visitors, and 


the gencral findings of Western surveys of R&D output. Yet this 


does nol necessa any mean that the collective picture is is a. afaithfut one, 


Is it possible to ‘eonduel ma. aysteninlic sliidics of Scientific 
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innzrefore: be as acc’ stable to ihe ecieniilic commniunity and s science- 
policy analystsin tic: sra-bloccountrics as theyarcinthe West? 


An Alternative Framework for the 


Evaluation of Research Performance 


The results of work at the Science Policy Research Unit over the last 


seven years suggest that.such evaluation techniques can indced be 
developed. In what follows, we describe ani approach, based on a 


number of ‘partial indicators’ of scientific progress, for making 
comparisons of major cxperin:ental facilitics {high-energy physics 
accelerators) and_of the scientists who use them, 

- High-energy physics excmplifies - ‘curiosity- -oricnted scicnee’ — 


that is, research funded primarily because of cg»: 


likely to make to the advance c¥ ‘rowledge * also leads to 
substantial cducational bcacits | = 5 fori or highly trained 
scientific personnel), i various tv; 3. technological ‘spin-off, nM 


and even to -broader political benefits. such as increased national 


atl and _ international bere ‘Yet these 


method aera various partial indicators of the scicilifie | progress 
made by users of different research. facilities: publication counts, 


citation statistics (total citations. citations per paper, and numbers of 


highly cited papers or ‘discoveries’), and extensive pecr-evaluation 


rankings of the relative outputs of the facilities. These indicators can 
be applied to matched groups of researchers | using similar_rescarch 


facilities and publishing i in essentially the sseie body of journals. In 


each of the three Big Science specialtics in. which we have used the 


method, a certain convergence between the results based on cach 


partial indicaicr. has been obtained; reasonably unambiguous 
conclusions can then be drawn about the relative contributions to 
scientific progress from each experimeriat facility: The main 
elements of the method are summarized in Table |: 
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Main Problems with the Various Partiat indicators of 
how their EATERS may he Minimized. Using the Methad of Conv 


TARLE 1 ! 


lhidicatars 
Partial m8 A 
Indicator Protten ' peeled aus + 
Raced on he Minimize 


A. Publication 
‘Conents 


1. Fach publicatinndoes not 
make anequal contribution 
In scientific knowledge 


Variationof publication 


inetitutionalcontert_ 


Usecitatinnsinindicate 

average impart of aresearch 

facilily’s publications, and in 

ifentify very highly cited papers 
he: 


spr td 
typesofpapers withinasingte 
specialty 


R. Gi fatian 


» Technical limitations with § Cl: 


Notaprobtem whendeaiing 


A aaly tis ( ay firstauther only listed, 
with arcsearch facility 
(b) variationsin names: ae = 
(e)aut bars with identical Check manisatly 
(4) erences: = 
(eine. omit SoveraRe of Notaserious sroblem far 
Big Science 
duringlifetimeol paper . wee an Bots EE 
os =unreengnized advanres) inn Nota probleniifcitations are 
’ ft zarded as an indicating nf 
impact. tather than quahty or 
3 ¢ tic importance 
4 ‘Malnefteeticitatiows 
3. Variations of citatic.n rate Choe! snatched ieealen 
witht ype of paperand facibatics pm oducing similar. 
: gprecisaley typesol papers within a specialty 
6: Scif-citation and“ia-hoiise™ Check cinpicicatly and aitjust 
citatina (SC and TiC) results ifthe incstznce nfSCnz 
NE varies Nelween gmups 
—_— — 
©. Peer 1. 1. Uselargerems entative 
Eveluatian _ sample _ 
facility ane competitors 2. Use vertal ratherthan _ 
> mayalfect evaluation writtensurvey inontertn 
2. Inilividualsevaluate pressevatuator ifaciver- 
SCH gence between capressed 


3. Confowmist assessments 
(e.gohaloeflect’) - __ 
accentuated hylack of knw. 
fecdlge onenniribitions of dif- 


ferent cexcarch facilities 
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opinions and actual views is 


v aluatnrsofcnn. 


fidentiatity Sp eeee: 


4. Check forsystematic 


variatianshetwcen different 
ernups nf evaluates = 


Use only indicators that yiéldiconvergent results: 


io) 
ERIC 


Irvine and Martin: Basic Research in Fast and West 301 
Our approach iii this study was to identify the mzin Eastern-bloc 
experimental high-ercrgy facilities; and then to compare thcir 


Scicntific outputs with those from the nearest equivalent facilities in 
the West. The data-base for the publication aid citation indicators 
has been carefully examined to assess the extcnt of any bias against 


Eastcrn-bloc science, and liciice (o overcome. as far as possible, the 
main problems.with previous use of such data, These-bibliometric 
data-are comple.nented by a survey of scientific opinion on the 
relative performance of Eastern-bloc. accelerators, involving 
interviews with nearly 200 high-cnergy physicists in both East and 


West; again, the results were carefully analyzed for any systematic 


bias. Although there arc definitely biases inall the oulput indicators, 


one can estimate their approximate magnitudcs. and so allow for 
their effects. The indicators, where they converge, can thei be used 
to draw what.we believe are reasonably accurate conclusions about 


the relative performance of Sovict and East European expcrimental 
high-energy physics. ‘There appears to be no reason in principic why 
stich an evaluation could not be applicd to other areas of Eastern: 
bloc science. 


Eastern-Bloc Accelerators aid their Closest Competitors 


High-energy physics has been comparatively well supported in both 


East and West. In Western nations, it has al times accounted for 40 


significant contributions would -be..made. to_ nuclear-energy | 


physicists began to emphasize the ‘fundamental’ nature of the field. 
and its potential contributions to other sciences.” In the Eastern 
bloc, such considerations have gone hand in hand with matters of 
national prestige. (Great effort. for example, was put into ensuring 


the Russian revolution.) As @ result, according to the senior 


Americag high-energy physicist. R.-R. Wilson, ‘the scope of the 


American and European programs’. an 
Table 2 lists the main Eastern-hloc accelerators opcrating from 
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TABLE 2_ 


Fastern-Bloc Accelerators and Their Closest Commipettiors, 1960-R2 


Accelerator DateofFirst = BeamEnergy Approximate Cost 
=~ ==" Opcecation (GeV) (USS$in)* 
PROTON >25GeV- 

Serpukhov(USSR) — - 1967 76 ~100 
CERNPS(W: Europe) 1959 28 45 
Brookhaven AGS(USA) 1960 33 uM 
CERNISR (W. Europe) 1974 mI) 75 
Fermilab (USA) 1972 sm 2sn 
CERNSPS (W. Europe) 1976 500 250 
PROTON <25GeV hut 5SGev nh % er 
Dubna (EF. Furope) 1957 10 ~307 
Moscow (USSR) 1941 7 ~2n7¢ 
Berkelcy (USA) 1984 6 34 
Argonuc ZGS (USA) 1963 13 50 
Rutlierford(UR) 1963 7 28 
ELECTRON >SGcV 
Yerevan ARUS (USSR) 1967 6 
Cambritlge CEA (USA) 1962 6 12 
7 


Hamburg DESY (FRG) 1964 7 : 

SLAC (Stanford) (USA) 1966 22 114 

Daresbury NINA (UK) 1966 5 12 
_ Comell(USA) _ 1967 12 12 
LINEAR €=<SGeV ae : 

Kharkav(USSR) 1964 2. No S107 

Stanford Mk 111(USA) 1982 1:2 : a 

Orsay (France) 1959 2.3 14 
c*e” COLLIDERS 


(USSR) (bh) VEPP2 | 1965 0.5 ~27 
(c) VEPP2M 1974 0.7 

(d) VEPP3 1970 3 
)rsay 2 
(France) (b). PCE. - 1976 1.8 13 
Frascati ADONE (Italy) 1989 15 -9 
Stanford® SPEAR (USA) 1972 4 ~24 
Hamburg DORIS (FRG) 1973 5 45 


\ctunh & (unadjusted for inflation) hased on official exchange rates: 
» Subsequently raisedin (OGeV.- = ee 
c. Estimates only, hased on the ensts of similar \Western machines. 
cost of anarcelera‘or inercascs approximately hnearly with encrgy. one weultlexpeci the Yerevan 
electron synchmiron with Five times the energy nf the Tomsk accclerator_tn -have cost 


approximately five times as much — i.e. ten mitlion dollars, if onc assumes an exchange rate of t 
rouble = (USdollar ek. 

e._Thislaboratory did operate asmatiercoallixtcrin the early 1960s, hit it was inscil mote asa test- 
hed for accelerator physics than as an experimental high-cnetgy physics recearch facility. 
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1960 to. 1982, together with thcir nearest cquivalent Western 
competitors. The. list. includes Virtually all the world’s largest 
accelerator facilitics."’ All the Eastern-bloc facilities are in the 


Sovict Union, while the main Western accelerator ccntrés arc in the 
US and. Western Europe. Since 1967. the largest Eastern-bloc 


facility has heen the Soviet 76 GeV (billion or giga-ciectron volts) 


most obvious way of identifying the main sets of competing facilities: 
Thus, the Serpukhov accelerator is most directly comparable with 


the slight'y oldcr and lower-encrgy synchrotrons. at CERN. (the 


Proton Synchrotron or. CERN_PS) and Brookhaven - National 


(ISR) and the ‘Super’ Proton-Synchirotrons at CERN. (the CERN 


SPS) and Fermilab in the US Mid-West. Similarly, the Dubna and 
Moscow proton accelerators can be comparcd with the machines at 
Argonne National_ Laboratory near Chicago*’ and Rutherford 


Laboratory in Britain, with theslightly older Bevatronat Berkelcyin 
California, and to a certain extent-with the somewhat larger but 


contemporary CERN PS and Brookhaven AGS: Among clectron 


machines, the Yerevan synchrotron had scveral virtually identical 


Laboratory in Britain.* The highcr-energy synchrotron at Corncll 
University and the lincar_accelcrator_at_the Stanford Lincar 
Accelerator Center (SLAC) in California also. provide useful 


comparisons. The Kharkov electron linear accelerator is very similar 
to machines operated _for many years by Stanford University (the 


Mark Til), and by the Orsay Laboratory in France. Finally; the 
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operated by Orsay, Frascati ae (in Italy), SLAC. ane 


DESY. 
How great. have. been the scientific_contributions. fram these 


i tous_Eastern- bloc. accelerators, ag frony. the principal 


en A recent t off cial US C tigress Report concluded (on 
the basis of rather limited evidence) that ‘to date, few significant 
physics discoveries have heen attributed to research conducted on 


Soviet accelerators.” How accurate is this assessment? 


Accelerator Outputs 
Scientific Publications 


ication counts are one of the mast frequently uscd, but also most 
tic, indicators of scientific output. Their tise as an output 
indicator rests on the assumption that scientists (in basic rescarch, at 
least) prefer to present the end-result of most research in scholarly 


publications. Not only is this thought_to be an effective way to 


transmit -informatian_ to the. scientific ic community, but it also 


community: furthermore, it acts as.a coriveitienit accounting device 
for funding agencies in ensuring that scarce resources have been 
effectivelytised. 

_ Because of the differingi sti itional, ; social and political | pressures 
to. publish in different countries, publication counts-necd to be 
undertaken with considerable - care (both technical and 


conceptial). * Tt niust be recognized, forexample, that publications 


do not all have an identical impact on the advance of-scientific 


knowledge, ancl that the same results are somctimes republished in 


different forms (often first in conference proceedings, and later ina 
journal). Morcover, in attempting to compare the scientific outputs 
of different research facilities, one must ensure that only ‘like’ is 


compared with ‘like’; for example; the Apparently greater 


publication output of one high-cnergy physics laboratory compared 


with a rival operating similar research facilities may be because the 


former has a large theory group attached (theorists teid to gaa 
rather morc frequently than experimentalists).” 
In comparing publication outputs, we have tried to ensure that our 
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scanned —-.in particular, national_physics journals and thasc. of 


individual Soviet Republics. Whereas such journals _probably 
account for.10 percent or less of all experimental high-energy physics 
publications in the West, the corresponding figure in the East is 
likely to be rather higher. This is especially true for smaller 
aceclerators such as those at Kharkov and Yerevan (the users of 
which sometimes publish in the Ukraine Physics Journal and the 
Proceedings of the Armenian Academy of Sciences respectively), and 


for Dubna, where visitors from Eastern Europe publish at least some 


papers in domestic physics journals. Tlie effect is probably greatest 
for Kharkov, where we estimate that up to 25 percent of papers are 
published in unscanned journals; but for the Serpukhov, Mascow 


and Novosibirsk facilities, the figure appears to be only 10 to 15 


percent,’ much closer to the..value for Western. accelerators. 


Although we have made some attempt to include such publications 
by using various other publication lists (sec note (b) to Table 3), our 
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= capa 2 Be es es TARLE3 snub ese See 
Numbers of Experimental High-Energy Physics Papers® Published in International 
Journals” doring the Preceding Two Years 


Accelerator’ 1962 1964 1966 1968 1970 1972 1974 1976 1978 19K 192 
PROTON 525GeV = Se. cet, feek ee. s 
Serpukhov - = os cs 129 63 109 123 1% 129 110 
CERNI , 37 157-211-215 23% 254 229-278 «246 «WR 67 
khaven AGS 17 SR 127 165 178 «150 153 93 60 Si 3t 


- = 2 = = UM RR SI OS 8 
= = = = ~ 5 107 176 182 179 16 


PROTON <25GeV Me ees ee ee Se 
Dubna _ o 4 23M 2 4 WO KM M 19 
Moescnw 7 1S 18 27 24 24 43 H% 2M 2 15 
Berkeley = 2 a2 1220 -F2 9S 83 oO OS 
Argonne Z.GS - - 4 5S 8B 97 192 6R 57 20 24 
Rathertard - =- 7 UW BY 6 3 mM mM 2 2 
ELECTRON >SGiev Se 

a 2 0 #7 7 #9 ti 0 

- UW 6 3 2 M 8 2 ee Lk 
Slamburg DESY - - tt 9 Mw WY 33 22 17 4 4 
SHAG 222: - = - 2) S54 R20 74 R33 
Daresttiry NINA > - - = 9 18 13 1 0 12) 5 
Crrnell - - - = 4 17 14 18 OR 3 
LINEARe"<SGeV : = 
Kharkov =) -> soe 6 Mm HW 3 BW WR 2 
StanfordMk I 4 019 22: «9k 12 6 2 4 - - 
Orsay 200 t of §$ 4 3 - 2. we 
e*e“ COLLIDERS Sb OS oe 
Nowosibirsk- _--- - - ' 9 Inv0 6 4 If 
Orsay ACO #DCI - - - ' 7 4 4 4 2 6 9 
Frascati ADONE - = = = S$  *#% 17 12 0 -5 
Stantord SPEAR - =- - 5s = = 4M M 2 25 
MamburgDORIS) -- —---~- = - + - ~- 1 0 3 1 10 


“th This publication list 
JETP Letters: Soviet Jou 


Reports and: on Hi ariqus laboratories, ant compilations such as 
Particle Data Grenp. An fs sics Literature 
(Berkeley, Calif: Lahorat 

publications list ex 
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East-West-collaborations. We estimate that this introduces a bias in 
favour of Serpukhov o¥ 10 percent or so,” which :lmost cancels out 
the first source of bias discusscd above: 


important.) Te ae ee ee 
Given these reservations, what do the figures in Table 3 tell us? 
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Although the annual output of papers from S as becn just 


Infersecting Storage 


imbalance also appears for the world’s main electron accelera‘ors. 


296 . 


re) 
ERIC 


291 


Irvine and Martin: Basic Rescarch in East and West 309 


Overall Inipact of Scientific Publications 


according to their relative contribution to the advance of knowledge, 
Scicnce-policy analysts have in_recent years turned to citation 


to the advance of knowledge, and hence_reccive less citations 
than a paner of similar quality in amore active ficld: A paper's 


lesscomplete. _ oe ager 

There are three. main sources of likely bias. First; there are the 
papers in our-list that ‘lose’ citations from articles. in journals not 
scanned by ISI. We have secn that up to 20 percent or so of Eastern- 
bloc papers are published. not in the cleven major international 
journals, butin the national physics journals of cither East European 


Republics (which are not). Citations from articles in these latter 
journals are ‘lost’.On the basis of the estimate that 19 to 20 percent of 
high-cnergy physics papers.in the Eastern bloc are published in 


unscanncd journals, we can expect a bias of similar magnitude in the 
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citation counts.“* * although i it is likely ¢ to be under. 10 percent for the 


more international accelerators (Serpukhov.and Dubna) since these 
carna larges fraction of their citations from Western journals, mae 
all of which: se this field at es are scanned | by. ISI. . 


publications were on. avCrage cited with the sanie nen as 


publications in (he list, the resulting citation bias against Eastern- 
bloc accelerators would he of the same magnitude as that inthe 
publication data” — thatis, 10 to 15, percent, and slightly more than 
20 percent in the case of Kharkov and Yerevan. However, itis clear 
from_ourclata on papers published in the physics journals of Soviet 
Republics that these are cited far less often than papers _in_the 
international journals (which havea larger audience and therefore 
reach more potential citers).* Hence; this particular bias is 
somewhat smaller, perhaps 10 percent or less. - 


A third possible source of bias, which is rather harder to estiniate, 


is that papers published én the main Sovict journals tend to contain 


less references on average than those iti equivalent Western 
journals. Full- -length Sovict papers typically contain around three- 
quarters as many references as Western articles, and ‘Ietters’ only 
half as many.” There are three. possible explanations: either 
Eastern-bloc physicists draw less heavily on previots cesults because 


such work has less impact on them; or they cite fewer references 
because their own literature is not so extensive as that of the West, 
and therefore contains fewer potentially citable articles; or they are 
more discriminating i in their use of references. If the last is true, then 


each Eastern-bloc cilation implics a. greater-degree of indebtedness 


than.a Western citation: This would suggest that, in comparing the 


relative scientific impact of research between East and West. one 
should ‘weight’ Eastern-bloc citations more heavily. Assuming an 


average figure (Sor both full-length articles and ‘letters’ ) for the East/ 


West ‘weight’ ratio of about two-thirds implics that ‘unweighted’ 


citation data_will be biased by 33 percent against the Eastern bloc. 


However, Easteri:-bloc research carns a substantial fraction of its 


cilations from articles in Western. journals, thereby reducing the 
effect toncarer 20 percent. Even this is probably an over-estimate of 
the residwal-bias, sinc the first explanation of the lower number of 
references in Eastern-bloc papers (previous . work having - less 
impact) is at least partly valid: Morcover, the bias should be 
considerably smaller for work of international interest (such as that 
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fall in the lower part of this range, and the smaller oncs in the upper 
part: This means, forexample, thatthe ‘crude’ citation figures for.the 
Kharkov accclerator should be doubled; while those.for Serpukhov 


or Dubna should be increased by just under a half. Stich biases 
should be borne in mind when examining Table4. -- caren 
_-Table_4 presents the numbers of citations achieved by cach 


accelerator over four-year periods from 1961 onwards as listed in 


{rom Serpukixov had the greatest impact bctween 1974 and 1976, 
these citations being to. experiments inthe early 1970s whicn the 


accclerator's first twelve years appcars to have been very much less 


approximately an order of magnitude Iess than that for thc CERN PS 


and the Brookhaven AGS, even after allowing for bias. These two 
Eastern-bloc machines have also made considcrably Iess impact than 
similar accelerators at Berkeley and Argonne, and probably slightly 


less even than the Rutherford Nimrod accelerator. Similarly, the 
impact of the Yerevan accelerator has been an order of magnitude 


- less than all -the other large electron accelerators (exccpt- for 


Daresbury). The various Novosibirsk machines have done little 
better, with an impact an order_of magnitude Icss than those at 


Stanford and Hamburg. The gap between the Kharkov accelerator 
and its rivalis somewhat smallcr, but this may be because in the West 
this particular encrgy-region was fairly thoroughly explored during 
the 1960s; and because by the early 1970s the interests of 
experimenters had switched elsewhere: 


--Overall, then, we conclude. that, although the Scrpukhov 


accelerator achieved a relatively high impact during, the first half of 
the 1970s, the evidence points to this having beenexceptional among 


Eastern-bloc accelerators. Even allowing for the various sources of 
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oe  TARLEA ~~ 
Numbers of Citations to Experimental Journal Articles 
3 : Published During the Preceding Four Years 


1964 1966 1968 1970 1972 1974 1976 1978 1980 1982 


PROTON 525GeV a ; | a ee 
Serpukhov = = = 139-273 -4R2 494 318 Sit 3m 
CERNPS © 375 1280 1327 1195 1153 1322 1135 1174 774 280 
BrookhavenAGS — 5971265 1350 1202 959 757 779 419 174 16 
CERNISR ~ = = = 271 R25 7% SON 443 528 
Fermilab - - = = = 19 778 1903 2622 131R ROA 
CERNS?PS - = ee OT 523 BD 

PROTON <28Gev =. Ss oe 
Dina 73,16 64 29 337 O43 
Moscow 40-50-25 23 M42 SRT 8H 
Berkcley~ 9 780 497 SI7 353 26 61 57 25° «16 
Argonne ZGS - Sd 248 399 485 467 3H 40S 312 113 

_ Rutherford - 2 1% 139 97 1233 mM 51 93 «47 

ELECTRON>SGev_ = 3 
Yerevan ARUS. - ot: c= 98 9 1 8 2R (32 
Cambridge CEA 33146 «19R «469 105 -47 15 - - LL 
Hamburg DESY ~ «8-304 279-245 RE RR 87d 3 
SLAC . -  - 9S 457 671 664 466 420 328 129 
Daresbury NINA - = = 9 73 -1 4t 19 45) 43 
Corel! - = = 87 72 105 1H R22 GR og 

LINEAR c” <SGev 7 : 

Kharkov: -- --- - - 5S ot St 7 228 %2t 19 29 
Stanford Mk 11 H9 178 220 tf 4 Ole 
Orsay. 7 40 % 18 7 13 6 - 2. 2. 

e i te COLLIDERS - = 2 Sic ae 

Novosibirsk — = = 400.27 29 MN 2 6 ID 45 


OrsayACO+DCh - - 6 119 107 -SR -24 21 34 © © «42 
FrascaliADONE - - = 5 63 170 231 49 -45 -2R 
Stanford SPEAR = = = = - 6 R66 826 33 304 
_. Hamburg DORIS - = = = = OW 387 462) 15 


Note: These igures were oblained by manual scanning of thie Sticvice Citation lider. 


bias,. their inipact on. the advance of knowledge in high-energy 
physics seems, in. gencral, to have been uniformly low compared 
with their. Western equivalents. Such a conclusion is not, however, 


the only. possible one: for example, citation totals may be largely 
determined by the scale of research activity at each accelerator, and 
because this is generally smaller in the Eastern bloc, this may he the 
reason for the lower citation figures: To examine this possibility. we 
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Againe the figures i in 1 Table 5 may he biased against Eastern-bloc 


accelerators: Of the three sources of bias discussed for Table 4. the 


second (papers: omitted from_ our faerie de is. as 


TABLE 5 
Average N Number of ‘Citations Per Paper for Journal Articles 
___ Published During the Preceding Four Years 
1964 (966 968 1970 1972 1974 1976 1978 1980 1982 


PROTON 525Gev ane - ae a = 7 7 
Serpukhov - - - 48 3.0 28 24 4:2 £2 413 
=. = -- (10:7) (6.4) (4:6) (3:9) (2.6) (2.3) = 
CERNPS_ 19° 3S) 3 260 2.30 -2.70«-2.20 2.2 2.2 1.4 
Brookhaven AGS 80 68 46 35 29 25 32 27 16 14 
CERNISR = = 5 = 129 4.0 74 54 44 «47 
Ferniilah__ - - - - 38 69 67 #73 3.7 27 
CERNSPS - - - - - - - 12.7 $2 3:8 
PROTON <25GcV aS Sas ee et Et _ 
Dubna OS Nt 09 06 06 09 13 0.7 07° OR 
= (0:8) (1-3) (1B) (1.2) (1.0) (2.3) (1.9) (2.1) (2.0) = 
Moscow OS 17 06 OS 06 06 07 OF 07 07 
= (2.0) (2.5) (1.0) (0.6) (3.5) (1.0) (1.0) 
erkcley 34°38 28 29 25 #23 63 #25 £7 20 
- 42 36 28 33 23° 21 463.2 #36 23 
Rutherford - 609 46 3:2 £9 2 it 1.20 2.2 1.5 
ELECTRON >SGev oe eet ee Oe ae 
Yerevan ARUS - - = = 8 OF 06 03 £4 45 
Cambridge CEA 390 54 39 3.0 33 $9 230 wo ss 
Hakone DESY - 16 61 #44 490 26 16 25 O07 046 
SLAC _ - - 48 61 49 43 30 29 32 2.7 
DareshuryNINA - - - 2527 23 448 #19 07 08 
Cornell - - - 62 23 34 #33 #23 Tk Of 
LINFARc” <SGcV Bae he SS = 
Kharkov = -. 08 027 O06 4143 OS 04 O85 Of 
Stanford Mk IH 36 #43 SS 38 S 2.0 - - - - 
Orsay n2008 16 09 OR U9 le - - 
ete" COt, LIDERS _ pe 2 
Novosibirsk ~ - 133 68 29 3:1 - 10) 6f0 0 6 
eae er een: = - - ~~ (2:8) (3.6) - (2.5) (1.3) =. 
Orsay ACO + DCI - - - 49°97 73 430 #435 #43 28 
Frascati ADONE - - - 100 3006 40 654 #6170 «20 68 
Stanford SPEAR - - = - = - 5 128 64 66 
Hombre ORNS 2S ee 


Nore: These statistics are based on the data contained in Talics 3 ae 4. The fignres in brackets 
refer to the average citation rates for just the articles published in Western journals. 
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is. < probably i in thic range Gciween 20 a 404 percent. However; the 
1 ) be. this large for papers published in Eastern-bloc 
journals; for: papers published in Western journals, it should be vey 
much smaller: 

Table. 4 shows that ‘papers | from play viniialy achieved quite 


however, very. inch greater, as can he scen from Table 3). Incced. 


for papers published in Western journals,.the 7 average Serpukhov 
citation nratec came te from. close to the initial rate for the CE RN ISR and 


fceawephaeeat fre rom m Europe and the US| were invalvedi in many of 
the carly experiments..In contrast, the average impact of the 
remaining two-thirds of Serpukhov papers appears to have becn very 
low indced. Asa result, apart from a brief period early in the 1970s, 


papers from the CERN PS and the Brookhaven AGS scem to have 


hada greater average impact than those from Serpukhov, despite the 
latter's considerable Boerne in (erms ee and more reccnt 
const ruction.— 


which also temporarily held a world lead) its users 1s exploited this 


advantage by carrying out at least a few novel and important 


experiments. flowever, this lead was short-lived, and, as Table 3 
demonstrates, only a very small number of papers was published 
during this period. Overall, therefore, even after allowing. for 
possible bias and the smaller scale of research activity, we must 
conclude that the average impact of publications from these 


machines sccins to have been low in world terms.* 
Highly -Cited Papers and Discoveries 


A major criticism of all the output indicators discussed so far i is that, 
while they provide comparative data on the total oulput of papers 
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papers which are included, but which ‘lose’.soime of their citations. 
The first is likely to have a negligible effect, sitice the morcim portant 


year.*” Since then, however, it has yielded a negligible nuiiber of 
highly cited papers, far fewer even than the earlier-vintage and 
lower-cnergy CERN PS and Brookhaven AGS accelerators, ‘ct 
alone the new machines at Fermilab and CERN. As for the other 
Eastern-bloc accelerators, their record is relatively poor, even after 
adjusting for a bias of 40 percent: Perhaps the best that can be said is 
that somc of them have come close to equalling the record of the less 
successful of the main Western facilities. 


Peer-Evaluation 


Our previous studies have suggested that, while indicators based on 
publication and citation counts provide essential information on 


303. 


TABLE 6 (continued) 


sta 8 1961-80 figuresadjusiedfor 1961-80 figuresadjustedfor | . 2 


= - 1977-80 1961-80-20 binsagainstE. Europe 40% bias against E: Europe 
On ee, ee 


ae ee Ch, ee ee) ee > ee ee | ee ey 


Lied Lied a2 az a> ; : 
Bee - 6 3 90 00 6 3 so m | 24% «ww go > 8 3% £60 
PROTON >25GeV 7 a : a a 7 : ia : ; : 
Serpukhov_ 0 0 0 o 19 4 2 1 32 6 2 1 9 45 4 I 
CERNPSI3 13 2 1 o 0s = 3 0 
Brookhaven AGS 0 0 0 o 03 2 4 2 
CERNISR MW 2 0 o 52 18 <4 I 
Fermilab _ 4 10 5 1 ous 39 #47 I 
CERNSPS 19 7 3 0 19 7 3 0 
PROTON <25GeV a : 2 2 = = = : - 
Dubna _ ) 0 0 i) 2 0 0 0 3 0 i) 0 l 0 i) 
0 rt) n) 0 0 0 0 0 0 0 0 0 I 0 0 0 
0 0 rt) 0 sz 7 2 0 
8 3 0 0 % 3 0 0 
Ruthertoru 1 0 0 0 10 0 0 0 
ELECTRON >5GEV 7 x = : : = 
Yerevan ARUS l 0 0 0 q 0 (i) 0 L 0 0 0 2 0 0 0 
Cambridge CEA - - <= <« 0 oO 0 0 
Hamburg DESY 0 0 A) 0 13 1 0 0 
SLAC 7 3 t 1 6 10 4 i 
Daresbury NINA a o 0 0 (i) 0 0 0 
Cometh 0 0 0 0 5 0 0 0 
LINEARe”<5GeV : : = z : © 
Kharkov 0 9 0 0 0 0 0 0 0 0 0 0 a 0 0 0 
Stanford Mk III - - - - 7 3 0 0 
- - - - 0 0 0 0 
: 0 0 0 i i 0 0 2 1 0 0 4 1 0 oo 
0 0 0 0 5 i 0 0 
Frascati ADONE it t) 0 0 6 2 t 1 
Stanford SPEAR 19 3 0 0 2 15 7 5 
Hamburg DORIS 16 7 2 0 19 9 3 0 


—_———— ee 
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Fastern bloc); and 18 from groups in three East European countries 
regularly using accelerators it the Soviet Union. The interviews 
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__ 2 


accelerator to sixth equal with Argonne, well behind the main 


performance of individial accelerators. The data also only repre: 
sent-the judgements of scicntists who have attempted,-for cach 
accelerator laboratory, to weigh up the relatively sinall number 
of discoveries or major advances for which it has been responsible 
with the very much greater umber of lower-level contributions.“ 
Such general rankings are not without utility, but more specific 
data are also required. We therefore focused in. detail upon six 


ef the world's principal proton facilities: the CERN ISR, PS and 


SPS. the Brookhaven AGS; and the Fermilab and Serpukhov 


o.. 
ERIC 


303 
Irvine and Martin: Basic Research in East and West 321 


synchrotrons. High-cuiergy physicists were asked t to wSscss these 


on atcn-point scale (10 = most suiccessfiil). distingnishing between 
their records in ec ang ‘discoveries’ andi in deal more precise 


sail the CERN PS was judged the most successful in tcrins of 


‘more precise measurements’. Again differences appear between 


the vicws of Eastern-bloc rescarchers and those of Western 


es TABLES 
Assessments (on a 10-point scale’) of main proton accelerators in ferms of ( a) 
‘discaveries’ (b) providing more precise measurements 


Assessments hy ttigh-Energy Self. Peer- Oscraft 

_- Whysicistsin: Assess. Assess. Average 

— ment ment Assess- 
United = Western Eastern ments 


_ _-States-—-- Europe--  Europe-- - 


HIER Serpukhov — 2.1(40.2)* 2.8(20.2) 4.2090.5) 3.R(40.5) 2.60.00,1) 2.7040.1) 
6 


_4(16GeV) [(a=SO¥ (ie 10s) (WEIS) (n=) (n= 149) 
CERNrS 6.0.3) 7.2¢40.2) 7.70 20.4) 7.10 40.2) 6.40.2) 6.9040. i) 
— — (28GeV) _ (n=SO)  (n=108)  (m=iS) (n= RH) (MERA) (n= 169) 
& NGS 9.6(£0.1) 9.40.1) R405) 9.5(20.1) 9.4201) 9.20401 
$ (BGeV) (n=104)- _(m=1S)- _(w=4R)- (msl) (n=1O9): 
8 CERNISR S.R(40.2) R-d(£0.3) 6.8403) $9202) 6.1( 41.2) 
a. (3141GeV) (m=1043° =(nelS) = (m=3A) (a= 133). spelled 
Fermilab: T.U(£0-2) RMAMI) TAL) TUCEOT) 7.20 00.1) 
“(SGeV) (i=t04) (WEIS) (n246) (NEN) (n= 164) 
CERNSPS S.R(40.2) 6.8(40.4) $.9(40.3) 5S. 2) $.%( 40.1) 
(S0NGeV) (n=104)- -(n=1S)- -(n=68)- (nelly {n= 16) 
<= Serpukhov 3.7(4£0.2) 4$.9(£0.6) 4.3(20:5) 3:5(£002) 3.0¢ 40.2) 
eee (n=102). <(n=IS) -(n=20) (i= 147) (n=167) 
E CERNPS B.A(EO.1) R.I0.4) RACE) KSEE) 8.5740.1) 
3. (n=103)) (n=IS) (HERB) (NEKI) (N= 167) 
§ Rronkhaven AGS ) 7.0(40.2) 8.1(£0.4) 7.1(40.2) 7.2(20.2) 7.20401) 
E (n=103)) (n=IS)  (n=47) (n= 120). (n=167)- 
y CERNISR 7.( 40.2) R-AC+0.3) 72340.3) 6.9402) 7.00401) 
Fy m=49)° {n=103)- (n=15)2 _(n=35). (nS13Q° (n= 167). 
E Fermitah 6.1(£0.2) S.A(30:2}) 7130.3) 6901.2) BO EMIY BIC EM) 
y: (i=49)) (R=IM). _(W=IS). (dH). (IZ) (A= 167) 
S CERNSPS R.0£0.2) RAFO1) ROLO3) R.2C40.2) R242) R.2040.1) 
= (n=49)) (n=103))— (n=IS) (Nn =67) (nN =1ON)-- (ne 1h7). 


a. in; = most successful, The assessments are hased on the judgements of high- 
encrgy physicists of the rctative-scientific outputs from the accelerators aver-their 
eitire period oLoperation tip to the linie of tlic iiterviews iiJate. 198 Hearly-1982. - 


b. The figures in brackets indicate the root-mean-square variations belween the 
asscssmicnis made by the different groups of high-energy physicists. piving some 
Approximate idea of the divergence of opinion within cach group. 

c. Sample size. 
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Eutopeans and Americans on the scientific a cnleuinaniee of thie 


Serpukhov accelerator. However, even the Eastern-bloc physicists 
pee Serpakhiv considcrably behind all five Western facilitics in 


An Ovcrall Assessment 


We stressed carlicr that. ifreliable Sener are tobedrawn about 


paitial anlicalocs used it in. the assessment, To what extent,then, do 


the pecr-cvaluation results accord with those based on biblionictric 
data? = - 


stouel utp of experimental pulcatons was less than that of all 


tlie alesai of‘ ‘more e precise 1 measurements’. As. ese ‘discoveries’, 


cyen ae a i citation: bias of up to. 30 ) percent. the sae. stil 


a inaghiiness ‘this j is again consistent with the peer evaluation 


results.“ The conclusion is that users.of the Sovict accclcrator failed 


to. capitalize fully upon its position between 1967 and 1972 as the 


world’s highest-cnergy accelerator — in niarked-contrast to uscrs of 
the Brookliaven AGS, and to a lesser extent the CERN PS; who 
exploited siew niachines that: between 1960 and 1967, held a similar 
comparative mast te — as, indeed, did Fermilab from ae to 
1976. 0. 

Similar. but even heavier, criticism can be levellcd agaitist the 
Dubna acce erator. This held a world energy lead from 1957 to the 
end of 1959, and it might have been expected to remain.a front-line 
machine through the 1960s. The publication and citation records of 

machines of roughly similar. energy (Berkcley, Argonne and 


Riitherford). show that, even after 1960, when -higher- -cnergy 
facilitics came into operation at CERN and Brookhaven. much 
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nt experimental work could still be carried out in this energy 


European physicist aptly summarized this conclusion: 
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Let us now analyze some possible reasons for this difference in 


scientific performance between East and West. 


Factors Affecting the Scientific Performance 
of Eastern-Bloc Accelerators 


the (ext to support our conclusions. - 


following form: 


The system for providing experimental facilities for high-energy physicists at 
CFRWN/at Brookhaven/at Fermitalvin Eastern-bloc tahoratovies is working so well 
iit Ht dloes Hat need Changing. 


. The responses suggest that CERN users were most satisfied with 


the rescarch facilitics available to them; 60 percent agreed with the 
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2 = sees TABLE 9 
Factors Explaining the Relative Scicntific Perfarmance of the Serpukhav Accelerator 
(perecntage of interviewees belicving this factor to have beeni important) 


= = 


Usersof Others - - All 
Scrpnkhoy Interviewees 
Acccicrator 


(n=20) (n=93) (n= 413) 


1 Poorlinadcquate detectors—c.g. asa 

result of lack of fast clectronics 65% 68% 67% 
2 Poorcomputer facilities - 55% R% 33% 
3 Accelerator poorly designed (atd- : 


fashioned) and inadcquate performance _ - 
(e.g. poorbeams) a 35% 17% 20% 


4 Acceleratorene rey net high enough - 


onty.a factor of two increase on previous 


niachitics. . <don overtaken 20% 16% 17% 

in fae problenis) and technical ces aes 

support O% 45% 4R% 
6 Poor scientific ic management at Serpukhov 30% 30% 30% 


7 Poorcontactsbetween research groupsin 
Eastern Europe, and between East 
and West 38% 26% 27% 
8 Poor competitive ethos in Enter irene 
— little scope for individual initiative, aes oe 
- Motivation problems, andsoon - 25% 23% 23% 
9 Researchers have fess experimental 
experience than in the West 5% 16% 14% 
10 General hureaucratic prohiems of Eastern 
Europeanscientilicsysiem—e.g. ees ees 
inflexible five-year plans 56% S7% - - ~ _SR% 


Note: These figures represent minimum values only; since they are faxed on a 
content analysis of answers to a gencral quicstion concerning the factors structuring 
Serpukhov’s scientific performance: If the interviewees had itisicad been -taken 
systematically through the above list of factors, we would undoohtedly have obtained 
considerably higher figures for the percentage believing cach (actor to have heen 


important: 


Co a 
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statement, aaleauly 36 percent disagrecd: Fermilab users were also 


relatively satisGed — the corresponding figitres were $4. percent aiid 
37 Percent respectively. The Ssitttation was rather different for 


MAAC of the ISABELLE project (to. build. a large praton- 
proton collider) were reflected_in the results — only 19 percent 


agrecd, while 70 percent disagreed: However, it was the tisers of the 


Eastern-bloc accelerators who appearcd inast. dissatisfied —= none 
agreed. while 94 percent disagreed with the relevan’ statement. So 
with what aspects af the. facilities 1 were they | par ticular ly dissatisfied? 
facility used | hy. non-Sovict researchers; as well as the targest 


Bastern-bloc accelerator. 
Table 9 shows thator one cof them main 1 problems tacing Se rpuk hav has 


sieidiary iiatvitinchtatiaii and all the main detccior facilitics were 


not ready until several years later; by then the larger Fermilab 


aceclerator was operating: The sane. siltation had afflicted the 
CERN PS a few years carlier.™ putting its experimental programme 
at an appreciable disadvantage to the Brookhaven AGS during the 
carly 1960s. A\t Serpukhov,, whereas the. construction of the 
accelerator itself, received priority. inthe national. plan, so that 
adequate resources and highly-skilled technical personnel were 
made available, the equally important task of preparing 
instittmentation and detectors seems to have had lower Priority: 
Respor: sibility for this cannotsimply be placedan apparent rigiditics 
in the planning system (sce factor 10, below), making it. wore 


difficult to_obtain priority for secondary equipment; sonie blame 


must also be attached to the scicnitific coinnitinity. As one East 
European scientist commented, 


Ifyou want to build: ancw machine. you musi ihitiki in advance about the detectors 
al pparates. The big mistake we niade was idl to think in advance 
ely. The res at, therefore: was that (he 


and experiment 
ahout how to use 1 he macht ec e effec 


Gik cit that a sitnilar i situation hada ariscnten ycarsearlier at Dubna, it 


iS perhaps surprising. that this mistake was not anticipated and 


avoided. Moreover, as several rescarchers. pointed out, the main 
detectors were not only late, but some at Icast (for example, the 
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attention of high-technology industry. High-energy physicists, 
thercfore, are not the most attractive of ctistomers: trying to mect 
their very exacting demands is risky, and they order only small 
numbers of each item. As one researcher put it: 


315 
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processing of data. arnittally. ‘thie Saiviet- made compuiters at 
Serpukhiov had rather limited capacity, and were, by all accounts. 


inrctiable. The Situation. -improved ears! in 1 the 1970s when, after 


computers. liuwever, the conipiiting capacity at Serpukhov 
remained limited as experimental activity increased, while servicing 
and using these forcign computers efficiently have led to problenis 


(for.__cxample,. authorization is. needed for forcign- -clrrerficy 
purchases of sparc parts or additional equipment). Morcover; 
although the central computing facilities at Serpukhov have becn 


considerably improved, most Eastern-bloc user- groups have rather 


poor home facilities, with the result that researchers can take weeks 
to complcte data-processing that would take hours in most Western 
universitics: 

A third factor is the intrinsic quality ofthe Serpukho seccicatGi 
itself." Hts design represented an extrapolation from. the carlicr 
CERN PS and Brookhaven machines, and. was therefore not 
particularly i innovative in world terms: The resulting accclerator has 
a poor repctition-rate and produces beams of (clatively. low 
intensity, two factors whicl have greatly constrained. _ its 
experimental output; and, becaiise of the computer. problems, 
interviewees reported that the beam was neither very accurate nor 
espccially well controlled. There have also heen difficulties with the 
external beams, especially the neutrino beam. Originally, it had 
been hoped to avoid at Icast some of the techiical problems 


anticipated with accelerator control and detector construction by 


ciicouraging tlic Participation of Western Scientistsin experiments at 
Serpukhov, bringing with them,. inter alia, some of the modern 
‘technology not readily available to high-energy physicists in Eastern 


Europe: However, the accelerator’s inherent limitations soon 
dampened the interest of Western scicnitists, despite the significant 
Western investment in cquipment for Serpukhov, by CERN and the 
French-Governnient in particular, Most of the Western uscrs of 
Serpukhov whom we interviewed reported being disillusioned by the 
accelcrator's quality during these crucial carly years when it was the 
world Ieader. 

Also contributing to this fairly rapid loss of interest was the fact 


316 


re) 
ERIC 


311 


Irvine and Martin: Rasic Rescarch in East and West 329 


with its higher encrgy and more atlvanced experimental facilitics, 
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Given that there appears. to be little inter-instituic 


mobility. and no formal retirement age for directors, it is perhaps not 
surprising (hat many interviewecs claimed that senior management 
does not always retain a firm grasp of recent research: developments: 
Such leaders cannot be in the strongest position to plan an expcri- 


mental j ‘programme to sl Ech anew y Feseurch facility, tor to decide 


decision- awaking’ in the hands of a sinall administrative élite had a 
stultifying effect, onc casitalty of which was the. motiv: 
rescarchers. This was scen as gencrally weaker than in the West, 
where morc aspects of scientific decision- -making are decentralized, 


paitilpals in determining. thie tlirection ofa labora s research 
eae 


the alee discussed above are Probably more, rather than ca 


severe. - 
Factors 7 -to.9 refer to. general problems faced by the whole 
experimental high-energy physics community in the Eastern bloc, 


rather than by the user-groups associated with one particular facility. 


The first concerns the relatively poor contacts both between Fescaich 
groups in the Easterii bloc, and between rescarclicrs in East and 
West. Sonic interviewees described the felationship between 
Eastern rescarch groups as being far too secretive, rather than 
(ruitfully collaborative, with. sharing of skills and equipment. To 
some extent, this may reflect difficultics in (clephoning, telexing, 


photocopying. re writing letters Bier or visiting. _ forcign 


ihe resulting fragmentation of Elfort has seriously Weakened the 


overall cf ascliventess of their rescarch programunes. 
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generally rather limited. (except for the more senior scientists), but 
researchers are not well integrated into the informal communication 
system. This is partly duc to restrictions on researchers circulating 
unpuhlished material to scientists outsid 


sts outside their institiite, particularly 


| ami convinced and have evidence thatthe question af opening up Scrpukhaw th 
the West was hotly debated within the Soviet aditiitiistration, with strong viewsin 
favour and against being put forward. The agement that fitially woii was that tlic 
disastrous expericnce with the Dubra synchrophasotron could ant be repeated: 
Western experience and technology were neciled to ensure an adequate 
experimental -output fram what was for years the world’s highest-ciiergy 
accelerator. The counter-argument concerned the ‘ideological contamination’ of 
Soviet physicists, In the cud it was decided the lesser evil was to open Serpukhov 
up. (interview, 1982) 


problems during their stay in the Sovict Union. 

_.. The user-community of Eastern-bloc accelerators was also seen as 
having a less developed competitive ethos than in the West: Onc 
intervicwee described the situation in the East as follows: 


While one must recognize the value-laden conuotation of the (erin 
‘competition’, as well as the negative effects upon individual 
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certain ‘Western inhorsinnies’® ® (which several East eae 
interviewees strongly condemincd), it does appear that,i 
young physicists have greater opportunities to devise their own 
experiments. bid for time on accelerators, and,.if successful, lead 
research collahorations. With this freedom and the higher level of 


motivation it engenders, Western researchers are more likely to 
have becn involved ina greater number and-varicty of ¢ experiments 
their Easteri counterparts. The fatter thercfore tend to have a 
lo wer level of experimental experience. (factor 9in Table 9), and thus 


fail to gain the tacit. knowledge that is so essential. i after an 


sudclenly changed i in mid- streutn. According tos some early mcrkere 
at ial UMA Eastern: -bioc TENCAtSHETS not only had a poor idea of 


stricture of hadrons) >but _also_.a- rather limited repertoire ‘of 


experimental techniques — adequate for carrying out the simpler 


and more obvious experiments on the Serpukhov accelerator, bit 
not sufficient for tackling br precise and ambitious second- sath ae 


Soviet acecieruion} is the dverinireaueratined apateii Tider whicl 
these facilities are operated’, no tess than 88 percent of our East 
European interviewees agreed to a grcater_or_ lesser extert; while 
nonc at all disagreed. One of the morc critical stated: 


abe in pa to avoid: “zing blamed for mistakes. (lntersi iew, 192) 


‘This, of course, isa somewhat extreme view, (A number of Western 
physicists were also highly critical of certain aspects of their ¢ own 


administrative systems.) Nevertheless, interviewees gave many 
as of the. cffccis, of ee tasks thati in the West would 


nunicrous authorizing signatures.” ” the siteatiod of research 
papers generally takes-longer because of the need to obtain. high- 


ievel authorization; activities involving several research institutes or 
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year plan foreveryihing, cvenhigh-cnergy physics. . A 
five-year harizon is the lime to ask for mancy for experiments, and things have io 
he planned rigidly thal far in advance. Flexible re-allocation is nol possible as in 
ihe West. (intervicw; 1982) 
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considerable disadvantage: ™ To improve the situation, considerable 
structural reforms appear to be necessary in the organization and 
adininistration of scietice — a task that-a number of Eastern-bioc 
governinents now scem to view as a priority.” 


Summary and Conclusion 


To conclude, we vee by ete what is know about the State 


finiitadiane 3 in existing SOuncES of ‘lata. anit proposed a 1 somewhat 


different methodological framework for East-West comparisons. 
We then uscd this to assess the relative scientific performance of the 
principal Eastern-blochigh-cnergy physics.accclerators. Even after 
a pli avowallets for the pee bias in the various 


accelerator have been ‘small in _comparison with the. nearest 
equivalent Western facilities. Of the main reasons for this 
discrepancy, some are tcchnical in _nature - (inferior: research 
facilities; scient { Iputers), buiev 


to he mercly. symptoms. of a. 1 wider. § 


malaise — both at the level of the rescarch laboratory (in the lirnited 
resources available and the way these are managed). and of Eastern- 
bloc science in general. 
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1. Noneder nf seni 


3._ The evaliiatinis techniques are described in 


ing Basic. Research: The Case nf the 
ice, Vol. 13 (1983), 49-86. 


Isaac Newtnn Telescope’, Social Siudics af Scien 
4. Sec the work 


Nuclear Rescarch (JINR) at Dubna in the Snvict | 


A Ilistory of the Collaboration berween the Europ 


Internatinnal Politics in-the US-USSR Scientific Co-operatinn’, Social Sirdies of 
Science, Vial. 11 (1981), 455-80. : ee 


1980). 493-803, cee de 

Il. A good exainple is §. Kassel, The Relatiouship-hetweeu Science aud the Military 
int the USSR (Washinginn, DC: Rand Cnrpnration .R-1457-DOREJARPA 1977), 

12. Forexample. sec the many studies refereneedinC: P: Ailes and F, W, Rushing. 
The Scien-¢ Race (New York: Crane Russak, 1982).- - ; . . 

13. Twn camprchensive and iniflicntial cniigré critiques of East European science 
are provided by 7, A. Medvedev. Soviet Science (Oxford: Oxtnrd University Press, 
1979): and M. Popavsky, Manijidlated Scicuce (New Ynrk. Doutiteday, 1979): 

14. Reasons far the limited access tn the dormal nicchanisms of informal Scientific 


56-399 O—86~—11 3 2 3 
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excliatige of research j nei (partcutaty catty reports an ‘work:i “im 1 | eogress ie: the 
Nbdesdlad of yap ale 


1976). WU-67. 
1S: Interlaced with cmigré acenints are the perhaps i iievitable occasional 


as and distortions. Indecd; these sometimes give rise to accusations af 
Nisin nztion -— see. forexample, the review of Popovsky (ap. cit. note 13) by T. 
Holden. ‘Emigré Paints Grim Pictire of Soviet Science’, Science, Val. 295 (7 


exageer: 


September 1979). 981-84, Such accounts must clea! yhe usedonly with pre: i caution: 


16. See C. Raysen (Chairman). Review af US-USSR Interacaderiy Exchanges and 
Relations ( Washington, DC: National Academy of Sciences. 1977), atid RL. Garwin 
(Chairman), Review of the US/USSR Agreement an GCa-ereration in the Ficlds of 
Science and Technology (Washinginn, DC: National Assembly of Scicnees, 1977). A 
useful summary is given in ©. R2 Graham, ‘How Valuable are Scicatific Exchanges 


with the Sovict Union? Science, Val. 202 bie October 1978). 383-9: Sce also 


rae was ako undertake jn which eminent Amicrican scientiae familiar 


“17. A st 
with Sovict work were asked ta cvaluate the health of their specialtics in the USSR: 
VR. Sce Kaysci:. op. cit_note 16, 10. 
19. Graham: on: cit: note 16; 383: 
-- 20. Seethe first chapter, ‘Internati 
Natioiial Scicice Roard, Science Indicators 1972 ( Washiiigtan. DC US GPO, ce 
These indicatars have heen extended in Scope in subsequent valtinies, and Computer 


ca Ire: _ who compiled the data under contract from tke Nationa Science 


eanipaiter eee ; 
21. -F. Narin and M. P. Carpenter,"The Adeq n Index as 
an Indicator of fiternational Scientific Activity’, Journal laf ie Aenerican Saciety for 


Information Science, Val. 32; No. 6 (November 1985). 430-39: - - 
22. Gustafson. -op;-cit. note -9, 31. An interesting recent study of highly-cited 
papers i described in the research nate fy F. Narin. J.D. Frame and Pt. P. Carpenter, 


‘Llighly Cited Soviet Papers: An Faploratory lavestigation’. Social Si. PRET CNC: 
Vol. a, wiT- N95 2 


more b sed against the Savict Union than the US Recane of the casicr access to 


dC arpenter: ap. cit. note 21. Scealsos.D. Frame andl). t R. ¢ Peeieeyn, 
‘Counts of ‘US and Sovict Science and Technology lournals’, Screntemerrics; Val. 3 
(19RE), 159-75. 


2s. Narinand Caspeitersen op. cit. note 21; 428-39: a 
- 26. See H. Schmied, ‘A Study of Economic Utility from CERN Contracts’: /EEF 


Transactions an Eniginecring Managenient, Vol. £M-24 (5977), 125-48. - 


27. This figure an the estimated world expenditure on high-encrpy pliysics comics 


from Table 4in- 1B. R.- Martin-aad J. Irvine, ‘CERN: Past Perfonnance and Foture 


Prospects — 1. CERN's Position i World Wigh-Fnergy Physics’, Research Policy, 


Vol. 13 (1984); 183-210. 
2. Sce the work cite: in nate J for details. 
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American, Vol, 242 Vanuary 1980): 39 $2, 1 2, 2.8 
-_ 33. Tae only exceptions -arc-ihree- high-energy eleciran-p% 
(PETRA at Hamburg, CESR at Cornell, and PEP at Sianford) 


32. RR. Wilson, “Tie text Generation af Particle Acceleratars’; Scientific 


itron_ colliders 
h heen 
excluded hecause they came into operation oily very recciitly, and bceaise, for 
PETRA and PEP there is as yet no-Sovict equivalent. , ; 
34. JINR is the Eastern blo equivalent af CERN, the European Organization for 
Nuclear-Rescarch at Geneva.. ; ee 
35. The Zero Gradient Synchrotron (Z.GS) at Argonne was built asa direct 
co.apetitor to the Dubta machine, with fuidcobtained during the ‘coldwar’ periadofl 
US-Sovict competition in the 1950s. This is discussed in Comptroller Gencral of the 
United States, Report to the Congress: Increasing Costs, Competition May Hinder US 


Pasion of Leadership in High Exergy Physics (Washington, DC: US General 

Accounting Office, EMD-8058; 1989): 15: : Ee 
3¢. An assessment of ihe scientific performance of these various electron 

aceclerators cx.be found in Martin aud Irvine (Minerva), op. cil. nate-3.- 

_..37. Sce Comptrolicr-Gencral of the United States, ap. cit. nate 35. Scie of the 

Westein. high-energy pliysicists whom-we interviewed shared this conel ‘Gite 


putting it rather more succinctly: ‘Toa first approximation, Sovict aceclerators do not 
exist. : = eae eat ae 
__ 38. Ait extensive discussion of the trehinical and conceptual problems of tsitig 


publication counts to evaluate scicitific performance is given in Martin and Irvine 
(Research Palicy).or, cit. noice 3. The szime paper alse analyzes the problems invalved 
in using citation analysis and systematic pecr-cvaluation data as indicators of relative 
scics tific contributiins: oot 

- 39. Sco D. Sullivan, D. 11. White and E. 3. Rarhoni. ‘The State of a Scicnce: 
tndicators in the Specialty of Weak interactions’, Social Studies of Science, Vol. 7 
(198M), 182: be 4 _: 

40. These figureson the outputs from cach accelerates cannot he obtained fram the 


o.. 
ERIC 


320 
338 Social Studies of Science 


1S! data-basc, which | 


not te research Facil 
greaily-reduces the ut 


naive Criglich. — 
43. Because the number of experiments that can he carried out ona collider at aiiy 
onc.time is very limited compared with the corresponding nuniber fora fixcd-target 


accelerator, colliders tend to produce fewer publications. 
44 Martin and trvine (Research Policy) op. cit. note 3: 
43. Note iat even a paper subscqucnily found to contain ‘mis 


taken’ results can 
sometimes have_a significant impact an the advance of kiiowicedgc if it stimulates 
research that might not otherwisc have becn carricd out, and its citation record will 
normally reMect this, : 


46. This assuine: that papers in journals not scanned hy 1S! contain roughly similar 


numbers of references to those-in scanned journals. 
47. This bias refers to the figiires on total citations and nunthe 


rs of highly.cited 
Papers. Of course, there would be no effect on the niiniber of citations per paper, 
asstiming that omitted publications were cited with the same average [requency as 
publications included in ourtist, f -Sho 


48. A further reason to assume that national and Sovict Republic journals are on 
average cited fess is that, according to interviews carried out in Eastern Europe, 
researchers scem to prefer te place ticit better papers in the international journals 
when ihisis possible. - os : boss 

49. This is basedon the assumption that excluded publications carn enaverage hall 
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_ 54. Consequently; there Sceiis to he little evidence from these figures to support 


the view expressed in some interviews that Eastern-hloc researchers tend to publish 


much narrower audicnee, and therefore stand far less chance of having a major inipact 
and being highly cited. This is particularly true, for cxample, in the case of papers 
published in the physics journals of Soviet Republics. — - - SS 

57. This paper reported the first indications of a rising total cross-section in hadron- 
hadron interactions, a resiilt later confirmed at the CERN ISR and one which 
contradicted the predictions of certain fashionahle theoretical models of the time. 

58. This docs noi-include a number of other interviews that were terminated 
prematurcly when it became -apparcnt- that—the: interviewce had insufficient 


knowledge to answer the questions satisfactorily, being unable, for example, to 


recollect correctly which accelerators were responsible for certain cxperimenial 
resulls. = Es ; : 

59. Where. for example, two accelerators were ranked first equal, they were cach 
given the ranking 1.5; where three were placed first equal, they were ranked 2(thatis, 
the average of 1, 2, and 3): 300d soon. : : 

6. The Kharkov and Novosihirsk machines were not included in this ranking 
exercise. ae ; : ae : . : 7 : 

61. These lower-level contrihiitions generally arise from experiments reporting 
more precise measurements of known particles and their Propertics witheut 
discovering anything new, 

@2. See note 43. £2 gee ge. |. 
_ 63. This is excluding the CERN SPS accelerator, which has only been operating 
since the mid-1970s, -- eo: 

G4. Further support for this conclusion. comes-from the-results of the attitude 
survey described in the concluding scction: None of the East Europeans interviewed 
ageced with tie statement, ‘Serpukhov-has been responsible fora significant niitiber 


of major discoverics compared with other major high-energy physics centres’: while 
94% disagreed. = = = ee - 
65. This was probably largely because of technical problems — sce note 7 1 


6A 
Sovict high-cnergy physics presented in Pages 100-02 of the Kaysen Report, op. cit. 


hile oi Rv Gisagreed.  - ee 
_ 68. See J. Irvine and B.-R. Martin, ‘CERN: Past Performance and Future 
Prospects — U1. The Scientific Performance of the CERN Accclerators’. Research 
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1960s and 1970s was that its Director was generally able. by diul of his political anc 

industrial contacts, to ensure that industry did supply the equipment iceded. Oiker 

Soviet laboiatorics were not sofortunate. 
70. Sce Schmicd: op: cit: nate 26. 


71. Accelerators at the other Eastem-bloc laboratories have 


lical use — this has provided a means of helping to 


and tice 


participation in CERN experiniciits = aliiost a thase invoived are from the 

Research fastituics, — Sek red as 
73. OF the East Faropeans. interviewed, only 1% percent agreed that ‘Palicy 

decisions relating to Soviet accelerators. arc Ucterntincd stricily liy scicnlificconsider- 


seats Father (han political ones’, while 59percent disagreed. 8 
74. This is principally because the researchers interviewed generally had a far less 
intimate _knowicdge of the other Easterit-blac accelerator faboratorics. with the 
exception of Dubna, which has increasingly moved towards muclear pliysies in recent 
years. and so is less of initerest to the conecriis of this paper tharis Serpukhov: 


_75. Much of the impetus for thiscame from experimenial workcarricdoutat SLAC 
and the parallel theorctical-cadeavours of the American West Coast high-cnergy 
physics commiitity. According to-certain-scnior. physicists-interviewed; CERN was 
also perhaps alittle slow torcalize the significance of these developments- Given that 
the major contacts between Serpukhov and the West were focused around the formal 
agrcenient wilh CERN, this perhaps constilutcs another factor underlying the delay 
hefore Easicrn-hine physicists began to appreciate the potential importance of the 
parton-modcl. : : cite 

76. For au interesting critique by a Western scientist of the way in which high-energy 
physics is currently administered and of the adverse effects of ilie intensively com- 
petitive pressures on younger researchers in particular, sce vacious articles by R. J. 
Yaes: ‘Physics Fads and Finance’, New Scicntist, Vol. 63 (22 August-1974); 462-63; ‘New 
Generation Mo 4 Today, Vol. 29. (Scpitcniber- 1976). 11; ‘The Time 
Has Come to Abandon the Pumps and Run for the Lifeboats — Reflectionson Leaving 
the Physics Professian to Stidy Medicine’ Bulletin of the American Vhysical Society, 
Vol 23 (1978); 21-22: ‘The Scicnee Establishment’, in R. Arditti, P. Brenian-and S. 
Cavrak (cds). Science and Liberation (Boston, Mass:: South End Press, 1980), 217-38, 
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something which deserves analysis in a separate article: ; 

80. In 1982. clic of tis Was invited -to Iccturcon the techniques of research 
evaluation in Bulgaria. Discussions with sciior officials revealed the cxistence of 
major worries in several COMECON countrics over the direction and cfficieney of 
R&D activities. 


Resarch Unit, Mantell Building, University of Sussex, Falmer, 


Brighton, BN1 ORF, UK. 
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en decline 


from John Irvine; Ber: Martin, Tim Peacock and Roy Turner 


Recent data on the comparative — performance of different 


nations suggest that the decline 


during the 1970s began to level of at the start of the 1 980s. However, 


rather than heralding the beginning of a revival, the evidence suggests 
that this is merely a temporary halt to the long-term slide: 


iris now simost conventional wisdom that 


British s=*- ce is in a state of crisis. Recent 


leading art‘cles in- Narure, -for example. 
have hada tone of desperation; with head- 
lines such as “University paymasters dis- 
credited —the British government's latest 
policy document on higher education is_a 
disgrace”, “More crisis for UK research”, 
and “Parliament's chance to act”.-A simi- 


lar, if more restrained, note of despair has 


ture 1985, which was submitted in June to 


financial straits of the dual Support sys- 
tem” (for financing research in universi- 
ties). 


330 


_ ln these and other analyses ofthe situa 
attention -has been focused on. inade- 
quacies of the funds and other inputs to 
research. Largely for lack of data. less has: 


been said about the effects of- financial 
constraints-on the outputs from science: 
We attempted to remedy this last year ini a 


ae for the pe ABRC ahh made use of a 


science. 
The evidence suge gested that, during 8 itie 
latter half of the 1970s, Britain's basic sci- 


ence declined appreciably, both in regard 
to relative output of publications and in 


their impact on the international research 


community. It was argued that the govern- 
ment may not be providing sufficient sup- 


| 
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port for research (or at jeast not concen- 
trating it in-the right places) to ensure the 
nologies vital to the long-term survival of 
British industry. --- aise : 

results of the Study (see New Scientist, 
1429, _ pp.25-29, November _1984) 
apparently reflects a growing realization 
that, despite all the limitations of scientific 


mulating_a coherent national research 
policy if Britain is to ensure that the scien- 
tific needs of new technology-based indus- 
tries. are_met- Trying to steer pub! 

funded research using only the mechanism 
Of scientific peer-review (or “the informed 
Prejudices of wise men”; as ‘it has been 
quantitative input and output indicators, 


is somewhat like-trying to run the eco- 
nomy using only the judgements of a com- 
mittee of senior industri rialists. the oo 

In what follows, we update the figures 
On Britain's performance in basic research 


Lamb (St Andrews University, UK) to 
examine government support for science. 


Perhaps the most readily accessible data 


on national research outputs are the pub- 


lication and-citation indicators contained 
in the National Science Foundation's Sci- 
ence Literature Indicators Data Base. This 


Indicators.report, and is based on the Sci- 
ence Citatior, Index produced by the Insti- 
tute for Scientific Information (ISI). (This 
database is the source of the material pre- 


alternative data series based on the 3,000 
journals scanned by ISI in that year, but 
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Table 1 UK word share of publications by Geldof research 1973-82 
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biased in. favour of English-language 
countries’) Third, since CHI Research 


share by 4 per cent over the 10 years: More 
encouraging, at least at first sight, is that ; 
the downward trend evident since 1975 


seems almost to have levelled off in the | 
two Years:since 1980, with physics, Che- | 


registering small increases. We-shall-re- 
turn to the question of whether this halt in 


picture is uneven. Physics and engineering 
have suffered-significant decreases of 2! 
percent and 20 per cent respectively, but 
biology has marginally increased its world 


the downward trend is likely to continue 
or is merely a temporary phenomenon. - 
__On the impact of British work on the 
international: -research community. a. 
father less optimistic picture emerges | 
from the citation figures in Table 2. It can 
be seen that the decline which began in the 
1970s has continued into the-1980s — the 
world share of citations for all fields com- 
bined has decreased by 15 per cent since | 
1973, with biomedical research. engineer- 
ing-and physics all recording falls of over 
20 per_cent: (Figures based on the 1981 
journal-set are relatively consistent with 
these values.) Furthermore; although 
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Table 2 UK world state of citations by field of research, 1973-82 


ae Percentage of total 
Field ae et eee, SS 
- 1973 1975 1978 1980 1982 


11.9 125 128 120 414 
117° 104 90 «685 = Ba 
112 102 115 103 9:8 
28 128 MS 13 115 
94°99 89 93 89 
1S 121 109 99 90 
87 94 73 #78 #74 
Physics. 76 78 -°61 67 $9 i 
112 10.9 10.2 97> 95 ---1:9% -12% © -x48K 
Table 3 Percentage changes in world publication share of major 
industrial countries by field, 1973-82 a 
= jes eee Resor 
____. Field Canada France ~=FRG Japan UK USA USSR world 
Biology +6% -30% -10% +24% +4% -5% —7% +8% 
Biomedica! aa , aw Guee te = = a 
__Fesearch ~10% = -27% = -+7% +64% ~-10% +5% =7% —5% 
Chemistry - ~17% = 17% = +11% +23% ~12% -6% -11% +12% 
Clinical medicine +4% 6% -13% +62% -9% -2% -17% +5% 
Earth and space --: ties Sea ee ey Ss pak fate aoa 
_ sciences +5% ~4% +46% +4% +10% -8% -9% 412% 
Engineering and she ME es, Be een = Sees 
~-technology © “IS% = +31% 0% +45% ~20% -1%--31% 422% 
Mathematics —28% 0% +15% +52% =5% ~19% +78% +31% 
Physics . -19% +7% +28% +38% -21% -9% -16% +20% 
All fields - ie ed ede ae Sou pe 
_ combined -8% 9% = +2% +40%--~10% -3% -15% +9% 


Table 4 Percentage changes in world citation share of major 
973-82 


industrial countries by field. 19 
Fist = 5 ad ae Restof 


--Field Canada = France FRG Japan UK USA USSR world 
Biology +13% -2% +13% +32% -—4% -5% —45% +6% 
Biomedicai = pee. 8 _ = a 
Sesearch 9% +10% +48% +83% -28% -4% =6% +3% 
Chemistry —14% +3% = +4% +59% -12% -9% -13% 411% 
Clinical medicine +2% +17% +3% +84% -11% -4% ~27% +6% 
Earth and space as ee 
Sciences -6% +9% +66% +35% -5% -3% -34% +12% 
Engineering and a Z — = 
technology -23% _+8% -11% +91% -21% +4% —51% +W% 
Mathieiiatics -10% +19% +3% +53% -15% -13% 45% +42% 
Physics —-23% +13% +35% +69% -22% ~15% -23% +37% 
All fields = —_- a Pan 2 == 
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there had been-aslight decrease in the rate 
of decline from 1:9 percent per acinum in 
the 1970s t0 1:2 per cent in the early 1980s, 
there is little sign in the citation figures of 
the levelling off seen in the world share of 
publications: — on eae : 
When the figures for Britain are set 


against those for the other main industrial- 


breaidown by field of percentage changes 
in national shares -of world publications 


over. the period. 1973-82. Japan has in-. n 


creased its overall share-by no less-than 40 | 


Percent and the-rest of the warid by 9 per | 
cent: the main losers have_been Britain : 
(—10 per cent). Canada-(—8 per cent). | 
France {—9-per-cent)-and the Soviet Un- 
ion (—15 per cent). although: in, the. last 


of -carlier- NSF/CHI data -was-that the 
weakest parts of British basic science are 


i 
i 
: 


share of more than 11 per centor less than 


7 per cent-in 1982: One can see that the 


two Cases. there are likely to be appreci- ; particu ularly -ala 


able -biases in the statistics since French | 
and Soviet scientists may: well be prefer- | 
ring to publish more in domestic journals | 
not scanned by ISI. ae a eee eae j 
__Such an effect would also give rise to an | 
element of bias (probably rather smaller) | 
in the figures on changes in national shares | 
of world citations given in Table 4: These 
suggest that Japanese researchers in_all 
cight fields have markedly increased their 
impact on the international scientific com- 
munity, with an overall citation rr that 
has risen by 6S per cent during the 10 years 
to 1982. Similarly. West Germany and the 
rest of the world: increased their overall 
shares by more than.10 per cent, with 


France also recording a- significant in- 


crease (8 per cent). Britain and the 

Union have again been the main losers. 
respectively. One interesting feature of 
the table is that Japan's targest gains in 
world citation-share have tended to be in 
fields where Britain has-declined most, 
biomedical: research, physics and en- 
gineering in particular: 


panies in both traditional and new indust- | 
rial sectors are becoming increasingly de-_ 
pendent on the results of research for 
maintaining innovative activity-and inter- 
national competitiveness (see Foresight in 
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Table S Strengths and weaknesses in British science: share of world publications. 1982 


Nuclear and Particle physics 4.6 Veterinary medicine = 13.3 
JPUUS = 46 General and internal medical 5 
Polymer research 5.5 “research _- 13:0 
Oceanography and limnology 5.6 Inorganic and nuclear chemistry 13.0 
General: mathematics 5.7 Probability and statistics 13.0 
Ophthalmology = $.7 Gastroenterology woe 129 
Radiology and nuclear medicine $7 i 
Chemical engineering 5.9 
Nuclear technology 6:0 
Solid-state physics 6.0 
Applied physics. - 6.1 
Agricultural and food science 6.4 
Cardiovascular research 6.7 


General biomedical research 


Computer technology 
Dentistry _ 


Gastroenterology 17 
Geology _— 16 
Immunology 16 
Haematology 15 


Dermatology and venereal disease 16 
Nutrition and dietetics 16 
Endocrinology 15 


2 
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witha a new sense of economic -realism en- 


British: science: especially those for more 
applied -scientific and engineering. re- 
search. thus give maior cause for concern, 
particularly when set.in the-context of the 
dramatic: fall-that has alrezdy occurred 
over the last decade fine Oneke 


gendered by the policies of the Conserva- 
tive government after its election in 1979, 


realized the need to increase their produc- , 


tivity and began to publish more research 
articles in international journals: aoe 
_In our view, however, a more platisible 
explanation relates to-the funds received 
by science. -Most. British -academic - re- 
search is supported by grants from the five 


research councils: with indirect support 
ovides 1 by the University Grants Com- 


penditures in constant prices (calculated | 


constant at about 25 per cent, xs has that 
of France at 8 per cent. Clearly, if Britain 
i 00 Comets more effectively in the inter- 


Fig. 1, the expenditure for each year being 
expressed as a ratio of that in 1977. Ascan 


be--seen, - research - council: 


strength of is underlying rescarh tafe 
structure. 


Optimism _ 


at what has h oaieat 1980 to deter- 
mine whether there is any evidence that 
the - tong term- decline in-British- science 
noted eaflier; the figures relating to the 
British world share a 


optimism.- From the- g Saeed Table. 1, 
one-can see that, while the overall share 
fell from-a peak of 9.5.per cent in. 1975.to 
8.3 per cent in 1980 (a fall of-14.5 per cent 
in five: years -or- nearly. 3 per cent per 
annum), the British share has _remained 
approximately constant since 1980. Does 
this- herald a renaissance in British sci- 
ence? : 

There are scveral possible explanadons 
for this Jevelling-off in the relative output 
of- British publications since-1980. One-is 
that British scientists; perhaps imbued 


using the official EEC research and de- | 
wards have been -plotied graphically in ' 


expenditure 
tite Me Rap enpetefarsad 1972, | 


between the-trends over time in ‘funding 
and- national share of - publications. It 
would appear that the 6 per cent increase 
in funding between 1970 and 1972 was 
reflected in the 3 per cent increase in Bri- 
tain's share of publications.between 1973 
and-1975; while the 7 per cent decline in 
funds in the five years-up to 1977 was 
associated with the sharp drop of 14 per 
cent in: publication share between 1975 
and 1980 (see Table 7). In other words, 

there seems to be a time lag of approx- 


#.| 
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* 1983 figure not yet available 


imately. three years ears between changes in 


tions have been normalized to 1980. com- 
pared- with 1977 for funding.) Similarly, 


there is some suggestion of a time lag be- 
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_Much more: research is ‘obviously 
needed to establish the existence -of-a 


direct-relationship between funding and 


publication output and also toexplore the 
reasons behind the fall in the British share 
of citations. (One reason, for example, 
may be-the widely- reported-shortage of 
front-line research equipment in wriiversi- 
ties and research council laboratories.) 
Nevertheless, it is clear that one can take 
little comfort from. what has happened 
since 1980 — the evidence. suggests that 


phenomenon... ee 2 ak 
Over the last year, there has been a 
considerable growth in concern in Britain 
with the state ef science. At the invitation 
of ABRC, the Royal Society is undertak- 


shares; with the rapid fall in the former 
between 1975 and-1979 apparently being 
mirrored in a similarly sharp decline in the 
latter between 1976.and 1980. - -. - - 
.. If there is ‘indeed an. association be- 
tween research expenditure and the Brit- 
ish share of world publications, this would 
Suggest that the levelling off in publication 


association were to continue to hold in the 
future, one would expect the drop in fund- 
ing since 1980 to be followed some three 
years later-by a sharp decline in the UK 


come -available in -1986: Furthermore, 
according to. the-ABRC's latest report, 
research council funds have continued to 
fall in real terms since 1983 (by perhaps 1 
per cent per annum) and,on the basis of 
government. estimates, will probably con- 
tinue to do so over the rest of the decade. 


If the past is any guide, it seems that the 
UK world share of publications may de- 
Cline at a somewhat faster rate over the 


foreseeable future. 


ing a study of the health-of British science 
which may provide-some-evidence on the 
extent -to which the - pessimistic concli- 
sions following from i of 
the type presented. above are consistent - 
with the views of the scientific community: 


tenuous, but it has led us to start increas: ' 
ing our links with. overseas universities 
that we see.as centres of excellence... . - If 


our suspicions are correct that the United 
King¢om is no longer up:at the front as it 
was, then that presents, in the long haul, 
very -dangerous problems~ for us.” 
(Quoted in The Guardian; 19th February 
1985, p:24.) As one of the Members cf 
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£20) a i= onaesmren Cooxea sassauaua Reauiied io 407) 
wo. ry e-~-sUK world share of publications normalized to 1980) 
. are @--eUX world share of Gigtions inormalized to 1962) 


\funding#/ | | 
Publications (citations? | 


Normalized 
yt 


world share: of 


1972 1974 1976 1978 1986 


that we face as a scientific and educational 
We acknowledge — support from 


well as assistance provided by Dr F. Narin 


f CHI Research: oO 
John Irvine. Ben Martin and Tim. Peacock are 
researchers at the Science Policy Research Unit, 
University of Sussex. Falmer. Brighton. BNI 
ORF. Roy Turner works in the Transport Studies 
Group at the Polytechnic of Central London. 


o.. 
ERIC 


io) 
ERIC 


Michiel Schwarz, John Irvine. Ben Martin: Keith Pavitt, and Roy Rothwell 


Abstract: This paper summarises the results of cam from ihe Science Policy Research Unit 


required. ~ == = ==: mann ee en te 
__This_peper reviews some of the methods, re- 
sulis_and policy implications of a stidy_on the 
Norwegian system of state support for industrial 
R_-& D, a study which involved a systematic 
assessment of certain major elements In. thal 
country’s R & D system. The rescatch was car- 
tied out over a 4month perlod ih 198! by a 


R&D Menageiseat IEG, 1982 


aol aes and discussing.in detail those 
findings that are_of generat relevance to current 
probicms in R & 1) management and research 


>. BACKGROUND AND rocus OF THE 


SPRUstupy ; 
The SPRU study was commissioned in Novem- 
a 
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pverament-firndedt system: for Wesdneaeia R&D 
i é lectively by industry and 
ale behind this was the 


need in post-war Norway to modernise the in- 


me SCHWARZ, ef AL. 


lack _of _accamntability. over the resulis of re: 
seatch. fn what: ~extentt is Borel criticism valid? 


dustrial base. much of which lacked expertise 
and capacity in R &.1). Over the 3 years of its 
existenec, NINE has- 
and complesity, estahl 
stitutes. (there_are currently 15, although some 
are more_concerned. with public. sector than in- 
dustrial R & Dy, and creating a large network of 
Commitices (some -27 at present) to determine 
and administer_the distribution of R & D re- 
$n_ 1579. the overall NTNF budget was 
imately 400 million kroner (USS - 87 
lion), seprescnting about 20% of total 
governmeni fending for R & 1 (approsimately 
2biltionkrones 
-tn- come with most indus‘rial countries, 
pees has alse initiated a range-of other me- 
chanisms-lor-the support of industrial R & P 
particular. the Industrial Fund and the De- 
velopment. Fand. Wence not onlyis-the overall 
fevel of governmental support for R & 1-2. part- 
fakirly. important ixsuc,_bot a0 is the pot le 
between the wirieus mechanixms for the sup- 
port of R&D. in particular, what proportion of 
R_& BP resources should _be aflacated to infra- 


sources? - a 
__In order to analyse and eaaluale Norwegian 


mmecparrcare fo the support ( OF industrial R & D, 


ence Pics. Research Unit-ta carry out the task 


Of providing these data and givitig an ‘outsider's’ 
assessment of certain key elements of the R&D 
support system |” 


_- The SPRU research focussed in particular 
upon three areas: 


@) the petlormance af a number of individ- 
tal rescarch institutes and the rel: 
_ ship. of their activities to industrial 


— needs; 
cc the sysictn. for allocating R & D granite 
to instittites and firms:§ = = 2 
(iii) the: averall perfe 


aivce_ af the NTNE 
system in sapporting R & D and indis- 
trial innovation. 


- ‘The assessment centred on four institutes and 
on their R&D ——s two industrial 


i Es Seca pees (SINTEF), Rogaland 


esearch Instituie (RF) and the Nc jan 
Defence Rescarch Establishment (NDRE) 
first three of these provide different ‘models’ for 


cflectively, critics ruaiag. for cxainple, to 
‘ageing’ probicms in research instilutcs und tn a 


collective rescarcli-with different organisational 
and funding patterns, operational benefits and 
problents. SI_is a large NTNF-operated insti- 
tute, while SINTEF. though dependent. on 
NTNF grants, is antonomous and closely inte- 
grater! with the Technical Un 


Defence — he included [otc 
three tnllective research nstita ics. so tha 
clenrly_ indentitied. ar he Roya'c ‘Commission did 
Not, “however, | tequest=- that. any... of _ the 
government-supporicd ‘branch institutes’ (which 
operate as collective rescarch os gonisations ins 
number_of specialised industrial sectors) be in- 
cluded in the evaluation.) 
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Collective 
Resrorch 
Institutes 


Kigere t The NTNF-hesed research sydem. 


2. Tilt CENTRAL ROLE OF COLLECTIVE Gi) grants ftom NTNF for fit 
RESEARCH INSTITUTES jects. often. e longer-term ond more 
One significant. characteristic of Noceczinn ic ae “pasic’ researi ears a eee . 
ee se Tcreeian te (ii) payment for R & D work cammissiance 
side aspect: ccs of R &_D,-in particular (i) the 
scientific. and technical infrastracture. based on 
one ay Feit and {ii _—— 
support for R_& 'D_progects in institutes anc 
firms (cf. OECD. 1978). This. together with the 
fact that the flow of NTNF funds to firms takes 
piace t0 # considerable degree, Bh collab. 
ora’ ¥ a 


ee example, 
funds: : transfer to that firm the results of a tong-term 


(i) general or ‘core’ funding from NTN; research project carried out by an institete: tn 


Table i Expencinure on NTN. roleed research 1879 
et re 


1, NTNF expenditure miltion kroner 


General funds to itinnes 185 2 (3872) 


Research grants—projects carried out in Indusiry 46.7 (12%) 
—oni her projects. a 


748 (19%) 
; 403 8 (100%) 
irom government departments, _—— — 

. “a 2680 


Len incrossing prp-ation 
development, The percentage devot moat REG hat s en 
Gavetopment, and 20 on). the devoted io P has consequent! 
fatten from: sbout 90°. in 1967 to 65% i 1979. Of thin 65%. fast over one ftth wae 
allocated to fame. 
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Soret 


Tconcomvation of support on hace fem in grocer! need af A & 0 mppart 


Porseanet quality end quantity of secrwin 
—eecbity of stall (0g. Watinnte rpeeesctrrs mowing temporarity to 9 fem to 


bmpinment project.rnerta) 
interchange of RD pernoncet betwen eniversiten ietnvtes and fema 
—devetopsnent ond maintenance of shies — 


Services — the range of services offered by collective rescarch inetitetes 


~-tesporabvenese 10 sae nae stpeciaiy of tral are meron: siced Brme 


of Remotes —indweertel “Teerenets’ show! range of support eanrices 
“eleamation en caren and ture nonce ol potential waar Of nwt from 


an <r amonnt to the work. ~ 
Ths comparatively heavy concentration tn fe- 


souters in reacarch insiitules gives sise to one of 
the principal sources of strain within the R & D 
support. system -namely, the  fecling among 
firms_that they receive | fxt-t00- amalla t- 


collective. research institutes, it ix 
some.of the majer characicristics-of collective 
fescarch-as-2. mechanism for implementing inno- 


wesaly ss ta os 3 which are ‘cormally 
the -NTNE_ system ‘and “this em pe 


in addition the-other type ‘of fawas—project 
grants - -43 distributed to ‘tema in divect_compe- 


tition with the institutes, since hoth sutmit _pro- 


oe firms (and especially smatier ones) fect 
es rdvaniage in. this 


example, 
only. 39% af project. finds Sllocated In 1979 by 
the Mechanical Engineering Committee went_to 
industry, white far the Electronics Committee 
the figure was_only-29%: Moreover, the actual 
sums _invelved: (7.2 and 4.7 million kroner 
respectively) were_in absolate terms extremely 
smalt conpared with the resources spent on 


3. DATA COLLECTION AND ANALYSIS 
In view: a central position of collective re 


inatiiutes-in the Norwcgian R--& D 


PPpr 
needs of the serviccs_offered hy instilutes. This 
tequired detailed information an. the. inputs 
(landing. stalling. -ctc). outputs (long-term re- 
search. commissioned R&D. work; support for 
industry in implementing new technology ate}. 
end overall mode of operation of the. iraiiiutes 
as well as data on the specific R & D require- 
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The Assrawmemt of Gocvrainemt Support for Industrial Research ee 
rated management, NING administrators, and 
targely through structured interviews with Insti- government official, = 


twte researchers and indvstrialists, with addi- ‘The main types of data used in oor assess. 
tema! information provided by institute iment are suramarincd in Table 3, these having 


Visirn Ge Mane Gel Kaci function end Seas ot ete sane 
aa wsege by fe 
ond chit loves ot inetintss = 
“tree A'S 0 imomnty Gitiererd types of 
views of firme on witty ae 
Of iratievies 


views of inetune- —bnalator (tiem or 
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tatie 4 information base lor instinsts sessment: : ites conduc ond & © projets eam 


hecti classified: ander the six types of critical 
iasxucs relating ta collective rescarch identified in 
Table 2 Table 4_gives detail of interviews car- 


ried out and R & D projects examined during 
the two months of field-work. In total, {77 inter- 
views were undertaken (typically lasting 14 to 2 


hours), ahowt fall of which were with institute 


initial cx etka 
enabling ux to identify the 
to focus, and eames 4 bok lee Seana 
the structured inlerview schedules used in a4 
field poe 


ee eet er od behead 


G2. 
hoe 
boc | 
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=by fie sire 
--- —= By 0&2. ttenalty of fees —_ 
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The Assesanen of Government Support for Industrial Research 7 
researchers, and industrial:managers were invi- complemented by the data on NINF funding: 
ted to make assccuments of the system for allo- in particular a 
cating _NTNF _project._granit, snd. the the Commitices for Mechanical _Enginecting 


Sowce: SPRU (1981). Tobe? SSS 
: ‘ The fl & D intoreity of 2 fem e definad aa tie tatlo of its expenditure on R &Ons 


Aigures were 0 to 2%, 2 $0 8% and over BX. 


seen, Institute A-exhibite a relatively Al. Inttitule B, process “developmen 


ny Firms and projecis for firms with a low | 
intensity. On the other hand. this institute has 
been more successful with product vel 

“with projects that it rather. than firms, has in- 
Hiated, and with work for high R&D intensit 
firma, The apparent reason for ihis is that Irate, 
tute A has concentrated its electronics research 


a) 


n firnss that its 


sely upervised, tend. be 
app Such 


tended. to involve only a small 
particularly ( employing former researchers 
from the institute. Indeed, most_of the efforts ti 

have been concentrated on assisting just one but it is less conducive to the crestion of tech: 
firm_which the institute hoped could he devel. nology suitable for commercial implementation 


oped into a major Norwegian-micro-clectronics in new products or processes. (Although _ we 


enterprise. Because of this coucentration of 
cflort. industrial projects in othet_arens-of elec- 
tronics, particularly for low R & 1 intensity 
firms, appesr to have received proportionately 
fess attention, and have therefore tended to be 


RE D Management (14, 1982 


have tented to focus on identifying the problem 
areas, it should nevertheless be noted that; itt 
the view. of most firms, Institute B has proved 
relatively successful in terms of providing R & D 
support for industry.) 
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Table_2. improvements required at one institute (A) 9s joociged try 
electronics tiers 7 
Nember of fenjects ascesved 30 
Areas of improvement ‘% of firms nungecting 
— a each improvement 
Beer ASD stat 32% 
Bettor project management 20°. 
Less. approach. __ 20°. 
Benet contact with inctustry 207. 
Keep to br ws yz 13s. 
Mare long term sesearch spciatixation 13% 
Keep to achechiins = - = ™ 
More temporay transies of sintt oa 
—-te-émplemnnt projects in industry Tt 
Staft with more iedustial experience Ww 
Betier marketing PR of institute - 3% 
Other improvements 13% 


Source: sPnu (1981). Tatte 30 


4 RESUITS: SOME LT USTRATIVE FINDINGS | 


Using these different sets of data; we were able 
to construct various performance indicators for 
nel e a eae institutes. and. the 


¢ views of ‘researchers and 


cts, it was aan to_ identify where 


improvements were felt to: he nected, both at 
vel of R & 1) management and govern- 


mental policies. 


The analysis of institute potformance_ dice 
heavily on é 


ewe 
: ‘ojects 
out_at-institutcs cnabled- us to identify 
the types of firms making use ‘of institutes, the 
natire_of the R.&.D support they received, and 
the industrial bencfits_as perceived both by in- 
dustry anil institutes. (We were also_abte to cs- 
tablish which types of firms made tithe or. no 
use of = facilities and services offered by insti- 
in conclusions could then be drawn 
pleted satisfactorily by institutes, and hence 
about rovements most needed. Table 6 
presents a selection of some of these data, show- 
ing -haw the success rate for industrial R =o 
Projects varies with-the type of project, the in 
tiator of the research, and the characteristics "of 
the -firm involved. In addition, from the-indus- 
trial interviews, we obtained an indication of the 
clentents perceived by firms as most problematic 
in relation to the performance of inslitutes. 


Table 7 gives an exaiiple of the resulis obtained 
here - . 

- In order to g00n to draw niore general ¢ con- 
Gusions at the tevel of KR &.1) management-and 
policy, iis important to cxtend. the analysis .to 
inchnle the determinants of success and failure 
in joint industry-mstitate projects, in particular 
institutional_and organizational factors. These 
are discussed below in the context of the critical 
issues_assuciated with collective rescarch ident- 
iftess earlier (see Table 2). 


@ ) Funding and programnie selection. Two 
main problems cxist here. (irst, the dual 
role of NINE as both a funding agency 
and _a performer of R & D has given rise 


reas of potential conflict 
between institutes and. firms, especially 
where they are compcting directly for 
the same funds. In the case of elcc- 
tronics,-for example, the long-term con- 


one Datel firm (sce ihe explanatory 
note to Tahle 6), has inevitably limited 
the support available for the devel- 
opment of this crucial area of new tech- 
nology _in other_institutcs {and Sirms). 
happened because the institute ‘was first 
in the ficld, there were then press 

ures to continue furcling its rercarch 


stall, This | problem « of fexioility in fund- 
iog programmes is compounded by the 
Jack of scope kw outside determination 
of the way in which institntes use their 
cure funds. More gencrally, 
‘because their project . proposa’ 

‘Compete with those from institutes, feel 
that they are at ‘a minjor disadvantage: 
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' zi in 
writing strong proposals and. are in a 
better position to know what sorts of 
proposals are most likely to be funded: 
Consequently, an application from_2 
nelag Feed Coe ee bee 
the proposed work és unpromising, but 
because the proposal is poorly written or 


the resources available are less than this, 


little 


ts made, and the return.on 


(tinuous inflow of new people and _iicas, 
and a constant development and evolu- 


these). SE, hawever, has. no such immedi- 
ale-source_ of people and ideas, and this 
may well be the main factor underlying 


R & D Management 11,4, 1992 


what was found tn be a rather less suc- 


y: _conducts 
he workl’s KR & D, 
substantial efforts mustobviously be de- 
voted to monitoring technnlogical devel- 
opments - overseax,- and dillusing the 
tesults to potential. users “in Norway. 
NDRE has found that by -far the mast 
effective way of dning this is not through 
Monitoring the — {echnical _ literature 


ather tess-successful in meet- 
ing the necds of firms for process devel- 
opments than they. have. with.- product 
devclopments “In addition, while they 
arc often able to help in. the carly stages 
of the innnvation process (carrying ‘out 
background _tescarch). the support they 
provide at the fater stages--for example, 
ae {ransiati of tcchnicat advances 
nin viahle production __protutypes---is 
talhet more limited. Other limitations 
include-the generally short-term nature 
Of the assistance ¢ provide, the ab- 


(tii) Services offercd. The ii 
his 


expericnce ‘amongst. industrial — se: 


searchers (with __certain — notable 
exceptinns),- and partly ‘rom _tieir rela- 
tively infrequent contact with firma A 
significant conclusion frnm nur axssess- 
ment was that such problems in dealing 
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apparcat. in the case of RE, “the ervaller 

repionally-hased research institute. 

Output. Fven when institutes Picasa 
ish; 


Often, cael 

firms, fack of subseqnent commercial 

success is primarily due to the absence of 
nh sta 


stalf with the nevessary 
e eto take over the technology, 


Product or process. In. other instances, 
responsibility tics to_a greater extent 
slitute researchers wlio, lacking 
ial experience. are unable to 
predice emeateh results. suitable: “for 


likely to be. achieved nigel finns are 
solved fions the outset --i.c. from the in- 


itial research stage wl 
Conliniic (ao provide Assista 


al pra. 
cess. Whe best way of cnsuring this is 
theough the temporary: ttansler of staff 
between firms and institutes. This what 
might he -desceibed as. “the first faw of 
technology transfer’ is: ‘the best_mech- 
_fov_ transferring, technology is 


through the movement of penplc’. 


the fatcr stages of the developincal 


a2 Avsesiment of the NTNF-hosed 
. Bee waar’ ae 


stitute rescarchers prewided the ms 
in assessing ihe operation and limitations of tte 
existing R & 0 support mechanisms, particular: 
ly these centied around NTN. They also 


Table 8 Overatt views of mechanical J engineering tiers 07 on the N NINF system~ ~-eitects, of ten air 


” Saaea SPR) (1981), Table 344i 


MC TH 5. Sc HIWARZ, EF At. 

y. 
fo retate. the vaising de grees of Sitisfaction to 
rial sector. firm size and firrn R &_D in- 


auical enginccring sector, small: 
with alow R&I 


of thein had fittle or 
and the suppust it- Similar data were 
Produced for. the electrical sector, and on firms’ 


"These perceptions of indastrialists could then 
he related to structaral problents inthe NINE 
lata_on patterns of re- 


R&D support s 
illustrate the approach 3 
ple: the perceived 


ally hetping only the large firms. 
(87% Of iistitiite sesearchers advanced this view 
coinparcd. with 22%. who disagreed.) Conver- 
sely. few (8% off researc 
institutes ac successful in meet z 
smaller_firms. “The importance of Giri tive in 
determining. the level ‘of _industeial assistance 
given hy the R_ & D support system is reflected 

A s larger firms _in 
NTNF resource allucation: firms with oyer 200 
employees Spel 83% of NINE 


i electrical engir 
ing sector. Chis is.c the fact-that small 
firms account for a _signilicant part of both these 
industrial sectors. The concliision is that small 


R ® 0 intensity 


= Site R n 0 intensity 

Overnt__ - Skntier terms - Laeger tems tow Modin High 
Views on ed “209. ematoyees) fy 20 emmsovers) <2 - Of sales) (2* 57s uf sates) «(> 5% of sates) 
NINE system: 8 net? ne? 
Good _(no. change need) oe 

satisle tory (bet coukd he ae a oo. = 
—dnoved) o- 55°35 22" 29". 43%. 
Not satisfactory oF ae =. 7 ae aaa 
_ highly aneatistaciory 60": ats 66% 43% 
No views om no - 
ie ~- Enowlerga | ot Sysiemn 2 407. a 22% 14% 
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Source: various tables and appendix 14 in SPRU (1981). 
firms apparently cannot compete for support on of direct_institute-industry competition 
equal terms with their larger competitors, —- for resources and ensute gicater Mexi- 
~ Finally, it is worth stecssing that perhaps the bili 
main strength of the tesults-on the performance 
of collective research institutes and: the overall 
R-&-D systcm was the degre i A 
cal data. The no i 


Progranmncs linked ta 


-- saGonaLR &B-necds, | 
(ii) The techilogy-transfer function of -te- 
; : 2 9 starch institutes should be strengihenes 
onstraics how researchers and industrialists particularly through-encouraging grca- 
held very similar views on the extent to which er interchange of_R & 1D -personnel, 
industrial R_& D projects have been completed therchy creating closer felations with 
satisfactorily _by the various. institutes These other_rewcarch centres and better links 
{wo groups also shared similat perceptions of between firms and institutes throughout 
overall performance_of the NTNF support _. the variourstages of R & D projects. - 
This was somewhat Icss true of their fiv) More_ special help for smaller. firms 
hon the degree to which collective research should: be provided in the way of fonds 
institutcs are succecding in mecting the R_ & D and services available from” the: col- 
needs of firms; institute re-zarchers were inevil- lective research institutes and N'TNF. 
Positive in their views on this () Greater -aiten 
than the firms, although even here the differ- likelv- industri 
ences were nat great. ci 


43 Possible Improvements in the 
Norwegian R & D Support System ; 
As_a__result_of our analysis, we were able to 
identify several_major. areas. where -improve- 
ments are “perhaps required in the Norwegian 
R &-D support system. The major conclusions 

emerging from our assessment are as (ollows: - 

(i) Organisational changes should be made keting, etc) to fi 
in the system by which NINE operates convert R & D_ © su 
both as a funding agency and performer innovations. This is especially important 
of rescarch, in order tn reduce the level in the case of smalies firms. 


R&D Mangenieia VIA. 1982 
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(vii) Greater cohesion is required. in qorcrn- 
rds_ industrial R&D 
CuK! invelve strengih- 
the links hetween the NINE 
aod other mechanisms for. eni- 
(rial innovation. and en- 
oe. with 


research focussed upon the scope for enhancing 
the clTectiveness. of the Ube old sysicm | for 
the inc 


an E 
the support of Norwegian industry in the future. 


3 at WIDER SIGNITIC “ANCE OF Of sity 


iven the: prog importance of ellective :na- 

development cant innovation 

an urgent neo (to devise im: 

lor_assessing the performance 

institutes) and: the ‘ef- 
{R& 


hereand the mcthodlogy on which it is shock 
may be of morc general interest tw Ay & We man- 
agers and policy-makers. 


t. & 
inputs and. outptits of the Norwegia ANTNE- 
based) R.& 1) svstem, and on the operational 
structures and p-actices which together. deter- 
niin 's overall: effectivencss. ALC the 
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evaluation of research performance undertaken 
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research performance, and has combined tradi: 
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burg (Director of SI) for providing detailed-com- 


menis.. on- an. cartier. -draft- of this. paper, 


However. the conclusions presented here are the 
respo ibility of the authors alone. 


Note: Copies of the SPRU Report can be ob- 


R&D Manogemen (24, 1982 


tained (for £5,000 cover postage and handling) 
from John Itvine {Science folicy Research Unit, 
University of Sussex, trighton BNI ORF. 
United Kingdom, 16 whom other correspon. 
dence concerning this paper should also be 
addressed. 
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APPENDIX 2 


CRITIQues ¢ OF lavine AND Manin: 8 Mainopoway 
AND A Repcy py MARTIN AND IRVINE 


European high-energy physics 


In he ny ans since the Siabishaent 
of CERN, the European Organisation for 
Nuclear ‘Research near Geneva, the lab- 
oratory has produced only two “crucial 
ies** compared with a dozen or 
States laboratories, whose 


total cost-is similar, and may-even be less. 


- This-is the bottom -line of-a lengthy 
balance sheet, with many other conclusions 
and observations, drawn up in-nearly two 
years work by social scientists Ben Martin 
and John ‘trvine,: of ‘the Science Policy 
Research Unit at the University of —_— 


and published today (6 Septembe he 
two-past report is certain-to be ir tial 
in--this- particular reappraisal of its 
membership of CERN. 


- The first CERN discovery categorized as 
“‘crucial" wasin 1973 (“neutral currents"*, 
a new form of weak interaction). By then, 
US! mbora vered 2 id 


s(u 
of: the aaa model), charge-parity-vio- 
lating interactions, deep inelastic scattering 
(which indicated that che proton really had 
components, now known to be quarks) and 
more. CERN, on. the whole,. has-been 
thorough but dull, say Martin and Irvine — 


CERN aN lacked “bzazz‘*. But the second of 


discoveries is very recent 


edi-. 
found: last year). -is attributable-to a new 
adventurousness at CERN and, according 
tothe review, ‘to a certain extent the 
balance_of. power in experimental high- 
energy physics between North America and 


Western Europe seems t to D have reached a 


d nd inne ate famed (or 
notorious) for. their Previous critical 
assessments of tie Jodrell Bank zadiotele- 
scope,- British astronomy. (notably: the 
Isaac Newton telescope) and the Daresbury 


Bu 


NATURE VON Tbe Sri aum es om 


electron - synchrotron {NINA.. But. their 
review_of CERN is more important as it 
may affect real policy i in two ways: first, a | 
British. scientific committee (sce box) is 
studying whether Britain should remain a 
member. of- CERN, given: the present 


@27-million price tag; and there are indi- 


improved approach to scientific assess- 
ment inthe CERN report is finding favour 
among British policy-makers, and could be 
applied to other fields. (A study has already 
been made of protein crystallography, for 
example, and delivered to Sir David 


Phillips, ¢ chairman: of | the Advisory Board 


basic tesearch (unds among the different 
research councils in Britain.) . 

But does the Irvine and Martin method 
work? Thetwo tesearchers* eatlier. reviews 


frequency. and peer ranking of tabora- 
tories, they ended up with too bleak a 
picture of what began to seem “targets” 

tather than mere subjects for research. 


There were mitigatine circumstances, and 


| indicators.: 7 


Bat Martin and Irvine's present teffor - 
two papers publisted in Research Policy — 
can hardly be criticized on those grounds. 
The CERN survey i is koopa eaten. and: the 


exampie, a anor described: their 
methcd as oneof “converging” indicators, 
thus begging the question of whether or not 
the indicators do indeed converge on a 
single black-or-white assessment of a lab- 
oratory. In the case of CERN, the indi- 
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the researchers, “the results based on Ape 
indicator nsed tc be carefully interpreted’, 
Interpret they do, basing their analysis on 
182 interviews with particle physicists from) 
the Soviet Union to the United States. - 


-_ High-energy physicists, even those at 
CERN, who have seen draft versions of the 
papers -describe the analysis as - fair, 
balanced and (mostly) accurate. CERN is 
admitted to have just missed the boaton 

many. occasions. The ttro quarxs_calied 
charm and bottom, which highlighted-a 
revolution in physics in the 1970s described 
by -Martin and Irvine.as “the American 

rn’? 


one physicist inter iewed by Martin and 
irvine as ‘’a tremendous faiiure’’. - 

~ CERN ‘even produced -serious exrors. 
One particle, the A2, was claimed — 
falsely, as it turned out-— to be split, or 
twofold, Creating a puzzle which abso bed 


in the United Prete (the ahs ie 
of Fermilab which caused equal conster: n- 


ation in the 1970s). Here the balance sheet 
is even. - 2 

- In terms of positive results, CERN‘s first 
big ‘machine, ihe 28 GeV proton ssn- 
chrotron(PS), though on line belore ihe 
equivalent alternating gradient. syn- 
chrotron (AGS) at Brookhaven, \ was no: 


where near as productive a at the he Bedinatie, 
Physicist Br zibute this, above all, to 
d 


lesign of high-energy 


rope. Then the 
intersecting storage rings (ISR), designed 
to collide PS protons with each other head 
on, was a brilliant technical success — Bui 
failed to produce revolutionary physics. (It 
heiped_in understanding the quark struc- 
ture of nuclear particles, but could have 
seen and not missed- charm and bottom.) 
sori and, Irvine suggest t the machine was 


than. four ytats after Fermilab in. the 
United States had creamed the energy 
region — and had to rely on tidying-up 
Which, nevertheless, i: did well, with 


resources. that were considerably greater 
than Fermilab’s), _- 

The lesson. if there i isa singleone, isthai 
Europe's laboratory was run (oo much by 
committee, whereas American laboratories 
Owed more to individual Nair. (Maurice 
Goldhaber at Brookhaven, R. R. Wilson ai 
Fermilab and-Pief Panofsky at Stanford 
Linear Accelcrator Center are singled out:) 
But Marcin cud Irvine ‘say that thiiigs are 
changing 6¢ CERN under director-general 
Herwig Schopper and that a more 
individualistic style has eneryed. : 

- (Nevertheless, CERN has commited 


Martin witl discuss the Biospects 1 ut ie 
and: at lease: one official involved in the 
British decision to support it has said he 


= 8 wishes he had had the report before hint 


y when hesaid yes. Robert W algate 
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POOR_MABKS_EOR_ENTERPRISE? 


V.F. Weisskopf 


entists Ben Martin 


The report of the social s¢ 
s of CERN. The Eu 
e” on 6 September 1984 under the title “Poor 


pean High Energy La 


oratory. in Gene 


Marks for ae . An observer well acquainted w. ith the American and 


European. activit ies in Particle. P Physics finds this report unfairly biased 


against th chieve “its of CERN. 


Started only in 1960 with a completely novel social experiment, when the 


international laboratory in Geneva was ready for exploitation. et no 


tradition in “big science” existed at that time in Europe anc . _ ne 


experience of international scientific, management existed anywhere. 


Nevertheless, within less than 26 years CERN hag developed into the leading 


laboratory of the world in its fielo: 


Why then do Martin and Irvine consider CERN’ performance 


disappointing? They concentratre their attention on sensational peak 
discoveries. in this respect CERN contributed only during the last 


decades. But science does not consist of discoveries that make headlines 


in newspapers. The bulk of science consists of a ‘broad. front of 


painstaking investigations, leading to ever deeper insights. They are the 


soil from which sme of those striking discoveries can develop: In this 


respect CERN fas mage many fundamental contribations to which Martin ang 


Irvine give only passing attention: 


It is true that some of the bad discoveries, in particular the 


s onal* discoveries. the violation. of 


vii rings (TSR) at CERN. Other 
: y, the Q-minus, and the t.. types of neutrinos. which CERN was. blamea 


.- 
s. 
a 
pes 
i 
«< 


for missing, were made in. the US before or at a time when research haa 


berely started at CERN. It is cert ainly misleading to use only the number 


of those well- -advertisea successes as & measure for the Quality of an 
institution. But even if cne insists On counting the lucky events of that 


type, CERN dig not fare so badly during the second half of its research 
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period. It discovered a-new form of radioactivity, the so-called neuvtre2 


currents, the well-advertisea quanta of the weak interaction, and probatiy 


even a sixth king of Quark. 


Hartin and Irvine ascribe the latest achievements to a recent 


change of Spirit due to the present Girector. True enough, Herwig Schopper 


vigorously supported the last phases of the conversion of the SPS to a 


proten- antiproton collider and the subsequent successful research. He 


deserves every credit for it. However; Martin and Irvine do not seem aware 


of the fact that one conversion of an accelerator and the reparation of 


Schopper s_directorship; The aaring Spirit and the technical inventiveness 


that hi ete ad to these achievements date from much earlier times. They 
are a Of of the opportunities given to individual scientists, which 


Martin and Irvine considered as too restricte{a at CERN. They also show a. 


steady growth of this institution towards its present world supremacy; that 


started long ago. i 
~ 


The comparison with the Fermilab in the USA is most unfairly 
presented. True enough, 2 a number Of great discoveries were made at that 


laboratory, such as the existence of a fifth kind of quark. 6ut it was the 


CERN accelerator. that was able to Produce sharp and intense muon- and 


neutrino- -teams with which the so-called structure functions of protons and 


neutrons were determined with an astounding accuracy. Moreover, the 


important °“EMC-effect* was discovered, which has lead to a new view of the 


role of quarks in the atomic nucleus. One of the main purposes of those 


eccelerators was the broduction of usable secondary. muon and neutrino 


beams. In this CERN was much more successful than the Fermilab inspite of 
the latter's four- ~year leaa: why Gid Martin and Irvine not appreciate 


these successes and why was this considered by them to be dull physics? 


CERN" $ contributions to the physics of nuclear structure are. 


almost completely neglected in Martin and Irvine‘ s report. CERN has the 


most advanced isotope separation device (Isolde) attached to its 


“synchro- -cyclotron where many nai ow unstable nuclei were discovered and 


studied. Recently a a low-eneroy source for antiprotons was constructed 


(LEAR) which provides unique -pportunities for the study of antiproton 


interactions: 
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-The success of CERN is based to a large extent upon a fact that 


was Surprising for people who have observed the development of physics in 


the last half century. America was assumed to be superior. in engineering 
and instrumentation as demonstrated by the development in the 1930's of 


cyclotrons, synchrotrons, linear accelerators bubble Chambers, etc. But 


the quality of the engineer- physicists at ak turned out to be at least as 
high. They succeeded in constructiong accelerators of unusual reliability 
and flexibiliy under the leadership of the late Sir John Adams. Some 

ign, but it helped greatly 


people ascribed this to an over-conservative 


in the exploitation Of the machines. for example, it allowed the 

luminosity oF tha intersecting storage rings to reach more than ten tines 

the design value and it facilitated the construction of. the previously 
Mentioned intense muon- and neutrino beams. Finally, it made possible the 
fast conversion of the 400 GeV accelerator into a proton-antiproton collicer. 


The success of this conversion Cepended upon one of the most imaginative 
Gevices invented ang carried out at CERN by Simon Van ger Meer. It is. the 
So-calleg stochastic cooling of the antiproton baam, a method that is going 


to be used in future colliders everywhere in the worlg, 


Furthermore, the CERN digidans cehysicisic leg by. Kjell 3 Jotnison 


constructed the intersecting storage rings at a time. when the American 
laboratories did not dare to take on this sifficult task: Indees, the 
first two colliders for protons were designed and constructea at CERN: The 
ee and the j@ proton-antiproton colliger in the SPS tunnel. With these 


continues the tradition by constructing the Satas electron- -positron 
colliger "LEP": 


The Hartin- Irvine. report does. not make much of these 


achievements; it 211 but ignores also tne numerous new instrumentation 


ideas that were spawned at CERN. such as the multiwire proportional chamber 


ovations are not only. 


ana the vacuum ultraviolet imaging device. Both ir 
significant advances ‘« Getecting particles, they have important 


applications ib me Fe Cine and in other sciences. 


High Energy Physics is an international endeavour, more so than ~ 


most other scientific activities. There is a constant exchange uf 
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physicists and enginters between CERN; the United States ahd other 
tgotries. I¢ does not make much sense to Consider the pre $$ in that 


field as « competition between two continents. ft is a collective 
achievement of ali pazti-ipating nations. The CERN effort has contributed 
essential ingredients. particles physics would not be what it is without 


CERN’S impetus, withou. its technical innovations and without its studies 
and discoveries. sien if some of them did not reach the newspaper 


headlines. 


Last 430 by no means least, CERN répresents the United states of 
Europe in fundamental physics, It became an active symbol of the spirit of 


Universities in Europe in scientific collaborations of unprecedented 
magnitude and complexity, frequentiy involving also Universities and 
Institutes of the ii,S., of Eastern Europe and Asia. In spite of the 
Obvious difficulties which such international collaborations entail, CERN 
has developed into the foremost High Energy Laboratory of the worid. This 
is strikingly confirmed by a recent American Panel report on New Facilities 
for the U.S. High Energy Physics program: in which it was stated: 

".:. The European facilities frequently provide a better level of 


How can such an institution be given “Poor Marks for Enterprise? 
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John Kt Krige and Dominique Pestre 


Question: Why do you spend money on high energy physics when you have so 
many other things to do? 


Answer: (1) According to Marx and Engne every discovery will have a a prectical 
application some time;- = tetra - = otesie ee 
(2). This kind of science presents the greatest possible challenge to engincers: 
physicists and industry, which is important tous; 

(3). Any civilization of any repute must be interested in the structureof matter, the 


origin of life, and the origin of the universe. 


(Response of a Chinese colleague toa question asked by v. Weisskopf.') 


The work of Soin. Irvine and Ben Martin, (IM) deserves serious 
attention. Ther range of their aS) iswide, they: address’ themselves 


not the same thing as quality as IM woah readily aeree. And aan 
there i is much to recommend i in Le % aAeselal its, € value is, we believe, 


followed by IMin presenting thei: ‘rosalts: We then goor on,in the core 
of the paper, to criticize several :<;ects of their methodology. In 
conclusion we try to assess the value which their work has, despite 
the limitations we have idétified. — - 

_Rather than trying to cover the wiole corpus of IM’s work. we 


base our discussion 1 primarily o ont their recent studies on bry} igh- sey 


Social Studies of Science (SAGE, London, Beverty Hills and New Delhi), Vol. 15 
(1985), 525-39 
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that is not our concern: If we are see mek anything, it is 
intellectual rigour: 


IM as Argument: The General Siroclure 


The plan generally followed by IM when presenting their results can 
be summarized as going through four successive stages. To avoid 


misrepresenting their views we rely heavily on quotations from their 
work. 


First Stage: A Critique of the Peer-Review System 


IM. stress the need ‘for. improved and more open mc ° sds of 


evaluating research. performance in basic science’ (LoS, 2). 


Typicaily, they give three main reasons for this. 
_ (a) There has recently been a levelling off and even a decrease in 


the science budgets in industrialized countries. Thus ‘existing 


fi nancial commitments must t generally aha reduced inorder to free the 


(b) With the concentration of resources in a few scniral: facilities 
having annual budgets ‘now running into tens or even hundreds of 


millions of dollars’ there-is a need ‘not just for accountability to 
Scientific peers; but for wider public accountability’. ‘One solution to 


this problem lies in the greater use of output indicators in arnt 


regulate the scientific system’. (EBS, 3) 


_(c) There has been an ‘entrenchment of particular interésts in 


decision- -making bodies’ reflecting ‘ previous patterns of resource 


distribution’. Instead of allocating resources on scientific merit; in 


the.‘free market’ of scientific ideas, a situation of. * oligarchy’ and 


‘rigidities’ has developed, which place an ‘increasing strain on the 
peer-review system’. (EBS, 3—4) 
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Second Stage: How to Evaluate Basic Research? 


IM’s method relies on four main considerations. First, ‘it is based on 
an input-output approach [...]. For basic science; some 
simplification is possible since one can concentrate on the primary 
output of contributions to scientific knowledge’. Second,_ the 
‘approach is institutionally focused’. Third; the ‘approach _ is 
comparative, with the added condition that one can only legitimately 


".. Third, the ‘approach _ is 


compare “like” with “like”’. Finally, it ‘involves the combined use of 
several indicators’ (EBS, 5-6) like publication counts, total 
citations, number of highly-cited papers, peer rankings, peer 
comments. . . Ifthe indicators converge (that is, when they ‘all point 


in the same direction’) IM ‘regard the results of the evaluation as 


being relatively reliable, and certainly as being more reliable than 


those based on a single indicator like peer-review’. (EBS, 9) 


Third and Fourth Stages: = 
rom Past Performance to Future Prospects 


‘the method is then applied tothe: ‘past performances’ ofthe facilities 


under study (stage 3). From this IM try to assess future prospects 


(stage 4). To this end they identify in the course of interviews with 
top : scientists ‘the factors that had structured success and failure i in 


performance i in the future I. > and. Ji any new factors that are 


likely to emeys=". (F 7S, 23) On this basis, they derive a set of 


comparative criteria i aseess the pi-espects for the various facilities. 


And they draw thefr> ws iusieras. 


Can Cne Reasonably =o t to Evaluate ¢ correctly _ 
Research Centres in ‘Big Science’ — to Improve Science 
Policy Deci-ions — Using Only Their ‘Scientific 


Contributicn to the Progress of Knowledge’? 


(a) ¥o avid fruit! -ss discussioii, we. want to stress that we do not 
doubt that IM say that culpuls other than contributions to scientific 


knowledge are important.‘ And we see much te recommend their 


decision to concentrate on this one aspect, particularly in their recent 
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outputs ‘secondary. reasons’ which -(in the case of high_energy 
physics, for example) ‘cannot explain why, over the last decade, 


considerations other_than that of scientific merit are of secondary 
importance in deciding whether to fund Big Science. The quotation 
we have put at the head of this paper makes the point. Results 


Siang 


scientific research and to compete-in the production of knowledge 
with their American counterparts. But a huge facility working on the 
frontiers of research in physics was necessary in Europe to retain the 
brightest. people in a situation of open (scientific and economic) 
competition with the USA: !n other words CERN had — and 
perhaps still has — an important socio-economic function to fulfil. 
Thus, without pretending to be exhaustive , some questions cannot 


acilities train 

people for industry? Do they train them in such a way that they 
have rather ‘unique skills?©) What would be the consequences of 
closing such facilincs down, or closing it down, when only one 
exists in a particular scientific fieldinacountry?...; 

— technological fall-out arid other economical benefits in industry 
(reduction of production ctists, development of new products, 
improvement of marketimage:.:), 

— ‘oiig-term indizect effects an military technologies:®§ 

— social role in testing new kinds of equipment, pushing them to 
their limits; 


— national prestige.° 


— manpower training (To what extent do big science | 
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least. nme physics. and § space ‘eames: hive a kind at ick 
analogous to that of military research in the economical field. - 


_ (c).We are not claiming that ‘non-scientific’ considerations, rather 


than the production of scientific knowledge, are ‘primary’ or ‘more 
important’ in assessing Big Science: Our point is, rather, that it is 
wrong to weight them aprioristically and, correlatively, that the 
weight in fact given to these different outputs depends crucially on 
the socio-historical context in which the decision, *¢i3 be made. At one 
extreme, if one has several institutes in the same country doing 
roughly the same work in a particular field, their contributions to 
scientific knowledge, as measured by IM, would conceivably carry 


the greatest weight in the minds of policy makers assessing their 
value. On the other hand, the ‘bigger’ the sciences and the fewer the 
number of facilities devoted to it, the Jess important this factor is 
likely to be. At the limit, if you have.a facility like CERN, the only 
one of its kind in Western Europe, it is inconceivable that one could 
close it down just! because its scientific pediormance. is is noi the s very 


does science which compares favourably with that done in. analogous 


éentics elsewhere in the world — though this would have been a 
ludicrous demand in 1960, for example. But this being achieved, the 
real problems begin: weighing a variety of outputs which cannot all 
be ‘measured’-in the same way; weighing them against the inputs; 
and finally weighing them against other outputs which may derive 
from similar investments in other fields of basic or applied research. 
In addition, these weightings must be done ona very long term basis 


because; while it is easy to'stop a big centre, itis a complex process to 


restart it. As.IM have shown, for CERN, it isonly after 15 to 20 years 
of existence that the laboratory began to compete with its American 
counterparts. And this would be truce for most facilities in ‘Big 


Science’. 


The Transition fe Hal ‘Past Performance’ to Assessing 
Future Prospects 


(a) The fit rst thing to notice is that the part- of IM’ s work dealing 
with future prospecis relies mainly, if not entirely; on peer-review, 


since..the potential scientific outputs. and the factors which 


structured, and are likely to.structure. the functioning of research 


facilities in the future are derived from interviews. This implies: 


ee 


362° 


r) 
ERIC 


357 


530 Social Studies of Science 


community, leading us to doubt whether they can. produce 

assessments substantially different from those obtained through a 

more classical panel system; meat fs Be. Be be on 85 
— that the results obtained using ‘quantitative’ indicators have little 


Accelerators’, as stated, for example, in the title of one of IM’s 
articles? We do not think the answer_is so simple. These 
qu2atitative indicators give at one and the same time an idea of the 
skills of the experimental. teams, of the impact of the science 


occurred in building the machines because of the attitude of some 
Member States in the past, of good and bad luck; and so on. In 
other words, these indicators give a global measure of the effecis of 


these ‘explanatory’ factors on past scientific output. To disen tangle 


them, to weigh their importance one against the other, IM need to 
refer to pecr interviews. Unfortunately, for policy decisions, it is 


this explanatory level which iscrucial, and not primarily the global 
measure. 


_ (b) Furthermore, when ‘projecting’ forward the factors which 
structured and are likely to structure. the future functioning of the 
facilities they study, IM tend to rigidify.them. Let us take an 
example. An interesting factor mentioned by IM — and one taken 


for granted by them and by the physicists themselves without careful 


assessment using historical data — is the opposition between ‘a more 


bold, speculative ethos of physicists in the United States, where the 
research system was regarded as no:‘mally allowing more scc:: for 
individual initiative’ _and_‘a_mor7:.solid, safe and conservative 
approach to their experiments’ by European high energy physicists. 
(C/I, 272) This opposition, when identified through interviews on 


the past performance of CERN, appears as one of the reasons most 


made in the European laboratory. As key discoveries are de facto 
considered in-IM’s papers as essential to assess the contribution to 
knowledge, the _American_‘values’ are connoted prsitively (and 
CERN is thought to have to ‘change’ in this respect:. Hence the 


expensive way of doing physics may, in another scientific context 


r) 
ERIC 


Responses & Replies: Krige & Pestre: Response to IM 531 


perhaps decisive advantage, even in terms of producing key scientific 


stich explanatory ‘factors’. First they must be studied critically in the 
light of historical data: If not, one is reduced to repeating general and 
banal statements. At this level of generality (see C/I; 271 »Table 9), 


‘objective data’ which IM provide do little, if anything, to place such 


decisions. on a more rational basis. What are we to make of 
Statements as general as 


Perhaps this information is interesting for the layman. But for 
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scientists and for policy makers it is far too imprecise. Depending on 


the importance. of the decision to be taken, they need _better 


documented and contradictory opinions. The ‘ecumenical synthesis" 


derived from. the views most commonly held among peers, and 
presented under the guise of objectivity. will be of minor interest — 
unless one is simply scarching for reasons to ‘justity’ 4 decision 
already taken. 


What about IM’s Remarks about the Rigidity of 
Peer-Review? 


To begin with, some precision is called for. When speaking about 


peer review are we speaking about peers asked to propose priorities 
between facilities in the same field (that is, comparable in terms of 
‘like’ with ‘like*) or between facilities which‘ produce’ knowledge in 
different fields (say, high-energy physics and biochemistry)? In what 


follows we shall distinguish between these two kinds of choice. __ 
_-(a) Concerni the 


developed by IM applies. What is more, as they say, the results 
obtained by asking peers to rank facilities have.been, up to now, in 


rather good agreement with the results obtained using ‘quantitative’ 
indicators. In other words, they demonstrate that peers, when asked 
to. compare ‘like’ with ‘like’ in a systematic manner, are rather 
fOUBUIG soca oe ee 
__Is this lest fact surprising? Contrary to IM, We would not lay such 


stress onthe ‘subjectivity™" of peer-ranking compared with the more 


the data (particularly citations) used by IM are themselves the result 
of scientists’ opinions about the works of their colleagues.-They 
embody competent scientists’ knowledge — though a ‘knowledge’ 


objective results obtained through quantitative indicators. In fact, 


not obtained in a systematic manner —of which institution produced 
key discoveries and which gave systematicand reliable results. These 


scientists are likely to have a good general idea of what the results of 
IM’s bibliometric measures are likely to be: In other words, the 
rather good agreement found by IM is not.to.be wondered at, and 
does not require special explanation. It isonly if adisagreement were 


to appear that a special explanation wouldbe called for. 
__.(b) Turning now to the second case — the choice between different 
fields — real problems arise. As Weinberg has put it, ‘tothe expert in 


oceanography or in high-energy physics, nothing see.’ - quite as 
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important as oceanography or high-energy phy-ics.”- Sut nere, the 


the budgets of institutes performing badly onscientific grounds. | 

- (c) It should be added that, when peers arc asked to make 
judgements of scientific merit across sections of the same discipline 
(physics as a whole for example), they are not always that 
‘unreliable’. A recent study. of the rankings made by a jury of 


assess proposals for basic research concluded (on the. basis of 
bibliometric data) that ‘the juties are competent: they know who is 


good and whois better’ #nd their ‘judgement has predictive valuc’.'" 

(d) Whatever its !ixs:'.a:tons, it does not follow that what IM do is 
of no interest. As they say, their approach ‘provides means to help 
keep the peer-review process “honest”’.'* However, it would be 


peer review system; and some powerful oligopolies have developed. 


assessment, in fact the limitations of the methodology they develop 
preclude them from making a constructive contribution to the issue: 


Further Less Important Criticisms 


the peer-review system “honest”’ (at least as far as these pecrs are 
asked only to give advice on the scientific performance of existing 


be made: 
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basically within these groups (EW, 304-06); their peer-review data, 
however, cut right across the distinction, ranking different kinds of 
facility on the same ‘linear’ scale; and so trea‘ing them asalike. Our 


_. Anyway, the decision about whatis comparable is not just a matter 
of mere convenience. Mulvey wrote recently that if SPEAR (the 
electron-positron collider at SLAC) ‘was responsible for the major 
part of the difference in score on which [IM] hang their case’; it is 
because this kind of machine was ‘the best type of machine for the 
physics of the 1970s’.'° We do not know if he is right, but if he is, his 
statement reveals that the definition of what is comparable must be 


clearly discussed in the light of each historico-scientific context. What 


is more, this context will not-be easily foreseen beforehand. As 
Mulvey put it, ‘in research of this nature, until the experiments are 
done, noone knows whichisthe best choice’. —— 
_ On other occasions also, IM’s criterion for ‘like’ with ‘like’ is used 


too loosely. For example, in the. same paper. (the. East-West 
comparison), IM explain clearly that the mechanisms of publishing 
in the Eastern and Western blocs are very different, and that the da: 


they produce are not readily comparable. That granted; their 
quantitative results derive what conviction they-have only because 
the. differences between East and- West are very big. They certainly 


add no rigour to what we already know or to the impressions pained 
by scientific peers.!° More fundamentally, would it-not have been 
more_ realistic simply to say that, given the great uncertainties 


surrounding quantitative measurements of publication outputs in 
the Eastern bloc, such comparisons have little to contribute to our 
understanding of the East-West difference? 
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(b) The notion of Partial Coni verging Indicators 


Of course Iw ’s response. to the above would be that Publication 


eachis imperfect o or partial in itself, the‘ ‘convergence’ ‘of the: separate 


indicators gives evaluative results which are relatively reliable. Our 


main_concern here. is with the significance of the claim that the 


quantitative indicators‘ converge’ among themselves, and with pecr- 
review. In our view it is misleading to describe the relationships 
which are involved in this way. Let us explain. 

_To dee with, what do the * i paar indicators — paises 


measures a ‘different facet of -the scientifi ic_performance of an 
organization (as compared with others), each contributes to the 
building of a global-and synthetical-picture of the organization in 
question. Thus CERN, for example, emerges as being at once 
productive (publication count), very reliable in its results, but weak 


in producing key discoveries (citation count). This_ picture. is 


obtained by adding together the results of more or less parallel 
‘indicators’. That granted, the question-of whether or. not these 
indicators ‘converge’ is ill-conceived, and beside the point. - - _ 
- What of the relationship between the results of quantitative 
indicators and of peer review? Here the notion of convergence can 
gain some purchase. But it is primarily a ‘convergence’ between two 
overall pictures, one built by ‘assembling’ the quantitative data, the 


other. obtained by interviews with peers. Here one can speak of these 


images as converging (or diverging). Yet even then the terminology 
is misleading, for it suggests that the two pictures of the organization 
are derived from_independent sources. As we pointed out above, 
hey-are. i not, and it is not surprising that, until now, the pictures 
»o:ained in both ways are roughly identical. 


Some Remarks on IM’s Rhetoric 


Our remarks in this section are inzpired by the recognition that IM’s 
papers are generally — and were for us— very convincing when read 
for the first time While all of us use certain. ‘ploys’ when writing to 


make our case, we believe that IM use a number of ‘rhetorical rules’ 


in a rather systematic way. We identify three in what follows. 


s 
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simplify. In this way, the reader's czitical faculties are disarmed. - 

The examples. of this ‘rule’ a:@ numerous. It operates, for 
example, when IM present the various kinds of Outputs one can 
consider; or when they stress that citaiton counts meusure the impact 


soon. In general these discussions are well done and are accurate and 
stimulating. However, the problem remains that, having discussed 
all the limits, IM do not always take them into account during the 


assessment, and then declare it the most important one. 


us a rough idea of the quality/importance/impact of an article, and 
that we will use them because citation counts are easy to handle 


general knowledge to strengthen your Gemonstration.  - —_- 
This rule-operates, for example; when one reads what we have 
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who practises history or sociology suspecis ihat peuple and groups 
tend to defend their own interess. that there are dangers when the 
sat people are both judge-and judged in a decision. Process, that 
lobtsing § : an important activity of scientists, and.so on. Ail this is 


dale “yy true. It does not follow — as IM show — that peers cannot 


be ‘honest’ when asked to rank facilities int their own field. However, 


common sense baulks at this conclusion, z and the refi nements in our 


understanding of the limits of peer review demanded by IM’s results 


are_not made. Received wisdom remains intact, uncritical and 


uncriticized: 


What have IM contributed? 
While what we e have. said above is consistently critical of IMs work: 
we wish to stress that there is much of value in what they have done. 
We. would like to identify the following important and helpful 
contributions theyhave made. oe 

(a) They have. collected, organized. and_ published a_colossal 


amount of information about the scientific performance of big and 


expensive scientific institutes. In so doing they Have provided an 


invaluable data base, containing both quantitative measures and 
scientific opinions, which no policy maker or serious student of such 
institutes can afford to overlook . 


- (b) They have shown that when peers are. assessing their own 


fi elds they can be reliable judges of scientific performance. And they 


have also shown the value of relatively cheap and semi-mechanical 


alternative techniques which can be used by social scientists and 


others. ‘external’ to Big Science to measure various aspects of an 
institute’ 's scientific ic : performance. - 
(c) Where choices have to be made ina fi aid wheie several similar 


research units within a country (ora group of f related countries) are 


compcting for resources, they provide policy makers with sound 


information for assisting a rational decision. 

_ (d). Finally, there are the ‘spin-offs’. Speaking forourselves, what 
they have done in the case of CERN is a source of i inspiration for the 
historical work we are engaged in at the moment. Quite apart from 


anything else it will help us ask pertinent questions of our archival 
documentation. 
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© NOTES 


__J-_This remark was made by V. Weisskopf at a seminar organized in conjunction 
with the Study Team for CERN History i Septereher 1984. Tac originatisona tape in 
the CERN Archives, Geneva. ; 

2. The core papers. we 


and Future Prospects — 1. -CERN’s: Position in World High-Energy. Physics’, 
Research Policy; Wal: 13 (1984), 183-210; (CIN, John Irvise end Ben R: Martin: 
*CERN: Past Performance and Future Prospects — II.- The Scientific Performance of 
the CERN Accelerators’, ibid. -247-#4; (Clil). Ben R: Martin and Jota Irvine, 


‘CERN: Past Performance and Future Prospects — II1- CERN; -rd the Future of 
World High-Energy Physics’. ibid. (forthcoming): (EW), Johi- 1: ine-and Ben R. 


Martin, “Basic Research in the East and West: A Comparison of the Scientific 
Performance of High-Energy Physics Accelerators’, Socic! Snaaies of Science. Vol. 15 
(1985) 293-394; (EBS). John Irvine and Ben R: Martin, ‘Evaluating Big Science: 
CERN 's Past Performance and Future Prospects’, Scientomerrics (forthcoming): 

_. 3. Our concerns are thus different from those of V. F. Weisskopf and J. H. 


Mulves,, whose responses toa report on the work of | rvine and Martin appeared under 
the heading "In Defence of CERN’ in Nature, Vol. 311 (18 October 1984), 599-600: 


_.-4. They have explicitly considered them, in fact, in their John Irvine and Ben R. 
Martin, “The Economic Effects of Big Science: The Case of Radio Astronomy’, 


Proceedings of the International Colloquium on Economic Effects of Space and other 
Advanced. Technologies, Strasbourg, 28-30 April 1980 (Ref. ESA SP-i51, 
September 1980), 103-16. eee SEPT! 1S 

5. Our work is availahle in a numberof internal reports produced at CERN in the 
so-called "CHS" series. See, in particular, D. Pestre; ‘Préhistoire du CERN: le Temps 
d'un Optimisme Raisonnabic, décembre 1350 — aout 1951" (Geneva: CERN, 1984); 
13-14.- - os: : : 

6 IrvineandManin,op.cit.note4. st 
-_7. The thesis of H. Schmied, Essai d'é¢v!uatio't des Effet: Economiques des 
Conventions Conclues Entre le CERN et I Industrie (Strasbourg. Université Louis 
Pasteur, 1977), isagoodexe™'2ofa study of such questions. While his method has its 
limitations, he is rather co: -ing wher showing that.in somic cases, the ratio of the 
economic wtilities produced in industries by the contracts with CERN to the amount of 


the contracts themselves is far from being negligible (31.6 in the case of precision 
enginccring; 17.3in the case of computing} or when he argues that roughly 80% of the 
total utility reported by firms derives from sales-to clienis foreign to high energy 


physics and nuclear physics. Sce also 2. Grinevald, A. Gsponer, L. Hanouz and P: 
Lehmann, Lu Quadrature du CERN (Lausanne: Editions d’en bas, 1984), 123-39. 


8. Consider, for example. President Reagan's recent ‘Star Wars’ speeches. See 


also Grinevald et al.; op: cit: soe 7, Chapter 1, for some suggestive remarks about 
military linkages. - eee |e 
9. IM themselves point out that, in the Eastern bloc, “matters of national prestige 
have played a major role in the funding of high-energy physics (EW. 301). oe 
10. Clemens A. Heusch, *US-Participation-at-CERN: A Model for International 
Cooperation on Science and Technology’, CERN EP Note, 20 January 1984, 11-12: 


I. See particularly John Irvine and Ben R. Martin, ‘L’évaluation de la Recherche 
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1406-16: a. 
12. A. Weinberg. ‘Criteria for Sc-entific Choice’, Minerva, Val. 1(1962), 159-71. 
13. W. Westera an. C. le Pair. “Evaluation of a Science Policy Instrument’; in W: 
Callehaut: S. E: Cozzens; B-P: Lecuycr. A. Rip and J. van Bandegem (cds).George 
Sarton Centennial (Ghev:t, Belgium: Communication and Cognition: 1984); 488-90. 


Fondamentale est-elle Possihie™’ La Recherche. Vol: 12: No: 128 (December 1981). 


14. Ben R: Martin and John Irvine. ‘Research Evaluation: Why? How*’: in 
Catlehaut ct al.; op. cit. note 13. 241-43: 

15. Mulvey, op.cii. note 3.- — 22 
- 16. Sce also the section of the same paper cniitled “Acceleratur Outputs — 
Scientific Publications’ (EW, 304. °s; -Ficre IM identify foor possible sources of bias 
int their data collection techniques used to compare publicaiion count. On the second 
of these. the difference in publication practizes. IN ure particularly convincing (EW. 
307). Unfortunately the authors do not estimate the poss:- as in this case: and in 
their cnclusion they quietly shelve it, givingan overall estir. bias which doesnot, 
take it into account (ZW, 308). 

17. Irvine and Martin, op. cit. note 11, 1409-10. 


John Krige and Dominique Pestre are both members of an 
international team currently writing the history of CERN. The 
_= latter is the author of Physique et Physiciens en Fiance, 


1918-1940 (Paris: Editions des Archives Contemporaines, 
1384). Authors’ address: Study Team for CERN History, c/o 
CERN, CH-1211 Geneva 23, Switzerland. 
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summarize.the essential elements of this methcd: Next, we state 
briefly what we believe to be ihc outcomes of IM'scf forts .o rest their 
method. Next, we criticize IM‘s claim as tothe ‘ alidity uf their 
method, and we discuss bricfly our opinion onthe application of the 
various indicators. In the second section we discuss IM’s choice of 
the object of their method: ‘Big Science’ rese’--’ jacilities. In a 
further section we comment on a few technica! ~ siems related to 
sitation analysis. Finally, we mention a number 0, sluable elements 
in IM’s work, and.we summarize our main criticisms: _ 

_..With. this article we respond in the first place to IM’s recent 
publications in Socia! Studies of Science," but in some sections we 
comment on their waik as a whole: 


Critique of the Method of Converging Partial Indicators 
Short Outline of IM’s Method 


IM’s method of convergin3 partial indicators fir comparing 
scientific contributions is described extensively in.their paper on 
radio-astronomy institutes? and moze briefly in their article on the 
Isaac Newton Telescope. Essentially, several ‘measures of science’ 


are applied, based on publication counts, citation counts, and peer 


judgments; these measures are assumed to be indicators of scientific 
progress. They are at best ‘partial indicators’, influenced by a 
network of interrelated factors, in which the size of contribution to 
scientific progress is only one: By combining the various indicators, 
and applying these to research groups, comparing ‘like’ with ‘like’ as 
far as possible, the influence of other, disturbing factors can, 


according to IM, be minimized. Then, only in those cases where 


Tests of ‘Approp. iateness’ 

IM claim that they tested the ‘appropriateness’ of their method in a 
number of empirical studies, and that these tests have béen 
successful. (See, for example, ‘NT, 54) What do IM mean by iis 
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term ‘appropriateness’: in other words, by what criteria were these 


empirical. tests performed? Thcy state that the appropriateness 


criterion of their method is the extent (o which ‘convergent results’ 
can be obtained: 

. Performing such tests, one should itemand that the method. is 
described clearly and applied rigorousiy, and particularly thit the 
phenomenon to be observed (that is, ‘convergence’) is defined 
adequately. - However, this concept of. convergence. is poorly 
deveioped. It is not operationalized at all. In fact. in their recent 
‘East_and West’ paper: !M_do not any more use the term 
‘convergence’ as such, but they add predicates like ‘some’ or ‘a 
certain’ to it (EW, 299, 322). Even worse, in their paper on the Isaac 
Newton Telescope, the various indicators do not converge; but this 
does not stop IM concluding that ‘the method appears to be capable 
of comparing the relative contributions of optical telescopes, and of 
producing significant results’ (JN7, 74). In our view, in order to 


perform these empirical tests adequately, one should use clear and 


illustrative techniques to indicate convergence. In the social 
sciences, especially in psychology, these ‘metric’ techniques are well 
known (multidimensional scaling, clustering techniques, and soon). 

_Asecond example illustrating the lack of rigour in description and 
application of the method relates to the concept of ‘matched groups’. 
One of the criteria for matched groups appears to be that the groups 
concerned pubuish in the same journals. East and West high-energy 


physics groups do publish in different journals and should therefore 


not be considered as matched groups. Consequently, according to 
IM’s methodological rules, the Eastern and Western groups cannot 
be compared. Nevertheless, IM do compare these groups. Another 
related point-is the incomplete coverage of journals. -In the East- 
West comparison, coverage of journals, especially for the East bloc, 

is_incomplete. In their Table I (EW. 300), IM state that 
incompleteness of journal-coverage is nat a serious problem in Big 


Science. But high-energy physics is Big Scie:ce! Why then do IM 
make such 2fforts to estimate the possible bias due to incomplete 
coverage? 

However, we would like tomakea point with respect to their work 
as a whole that seems even more important. One should distinguish 
between the validity and the applicability of tz..: method. ‘Validity’ 


refers to the exter* .« which the concept of ci’ tribution t to scientific 


progress, operationalized by a set of curverging indicators; 
coincides with the concept of scientific progress which the method 
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claims to measure. ‘Applicability’ of the method, on the other hand, 


refers to the conditions under which the method can be applied in 


order to provide valid results: One of the conditions for applicability 
of IM’s method — in our view the strongest one — is the occurrence 


of convergence with respect to the various indicators used: The tests 
performed by IM partly do — and in-principle can — examine the 


applicability of the method, but they do not and cannot provide any 


results relevant to the question of its validity: In particular, their tests 
can never provide results that invalidate their method, or even 


question its validity. If divergence should be observed in some cases, 
this result does not have any implications for the validity of the 


method. According ‘9 IM’s methodology, one should then simply 
conclude that apparently their method does not work in such cases. 


So the question of the validity of their method has, in fact, 
disappeared in IM’s studies. We will now discuss this problem in 
more detail. 


The Problem of Validity 


The question of whether IM’s method — if applicable — actually 


measures what they claim to measure, is nevertheless extremely 


important. In our view, an i nconsistency appears in their defence of 


the claim that convergent indicators do provide reasonable estimates 


of contribution to scientific progress: From the statement that al! 
indicators are influenced by a network. of interrelated factors of 
which the size of contribution to scientific progress is only one; we 
can_conclude that if disturbing factors are minimized, convergent 
results should be obtained in_ combining. the « srious ‘partial’ 
indicators. However, IM reach precisely the opposite conclusion: 
they conclude that if convergent results are obtained, the disturbing 
factors are (pri yably) minimized. They do not provide ary evidence 
in favour of their conclusion. On the contrary, they seer ic, provide 
evidence against it! ; Deena 

_-In their. papers, a number of Processes or me '22resees are 


described that might led io convergence of the vari. ::: #2’ - ators, 
without any guarantee that_disturbing factors affecting these 
indicators are minimized: First; the applied indicators appear to 


influence-each other. Second, it is a plausible assumption that 


existing disturbing factors will affect several indicators in the same 
direction: We illustrate this by giving two examples. In the Isaac 
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Newton Telescope paper, we see that this telescope has produced 
very few scientific papers compared tothe US telescopes considered. 
Differences of the same order of imagnitude are observed for 
citations (impact). IM write: 


What factors give rise to these apparently very large differences in impact been 
the British and American telescopes? Undoubtedly the main factor «5: the 
substantially lower publication rate of the former. (INT, 67) 


In the Radio Astronomy. paper, IM state that the citation rate of a 


scientific paper is determined partly by its impact, and partly by 


social or r political m measures s that, amongst others, ‘can be expected to to 


occupying ¢ different cognitive and <ocial locations RA. 71). From 
the discussion on peer evaluaticn it appears that ‘social and political 
pressures within the scientific community’; and ‘diversity in the 


cognitive locations of peers’ (RA, 73) affect the validity of peer- 


evaluation measures as indicators of scientific progress as well:. 

In our view, using their own conceptual framework, one should 
conclude that convergence of IM’s partial indicators is at best itself 
partial indicator of minimization of disturbing factors. As a 
consequence, convergent results provide at best a partial indication 
of scientific progress. 


Our View 


We think that quantitative measures — both bibliometric indicators 
of output.and impact, and ratings of peer judgment. as well — can 


provide valuable empirical data for generating and testing 2 questions 
and hypotheses on various aspects of research performance, rather 
than an instrument. claiming-to measure ‘the contribution to 
scientific progress’. We would iike to iiustrate this. view. At least in 
their two recent Social Studies of Science papers, IM actually seem to 


use their quantitative data in tests of hypotheses — «lthouzh they do 


Not state this eynlicitly, and although they still stre-s the importance 
of their's. © sonverging indicators. For instance, in thetr ‘East 
sti" Vest" papi , they essentially test a number of propositions made 

»gré scientists and (bodies of) Western scientists on Russian 
scientific. performance and science policy, presenting consistent 
empirical evidence thats- msto confirm most of these propositions. 
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In thei r Isaac Newton Telescope paper, somewhat contradictory 


claims of groups of English Scientists on research performance and 
research management with respect to the telescope are examined in 


_ This last. example, especially, illustrates the particular use of 
quantitative. measures whi-h, in Our opinion. is most valuable in 
research policy: namely, the application of quantitative indicators as 


in allocating funds. The outcome of such a test is a conclusion about 
the extent to which claims of ‘local peers’ are confirmed or re jected 


analysis. 


Critique of the Choice of the Object of Evaluation 


IM deal wich a rather high (and, at the same time. rather curiou'* 
aggregation lev-!: not a research centre or a_(very) large instituze , 
but a facility, an accelerator, within a research centre. This choice of 
aggregation level has important policy-relevant consequences. As 
soon as, at ‘east in the West, a specific facility — like an accelerator 


— 4oses its position at. ‘the front’, scientists will move to other 
facilities. It is_not then any longer interesting to follow the 


‘p<rformance’ of the obsolescent accelerator. It would, however, be 


relevant to policy to follow specific (groups of) scienti-t- “<> - at 
least again in the West — the participating univers’ “e 


groups form the basis of Scientific progress and ar an, 


the most interesting (and therefore most policy-iz evel’ to 
evaluate. oe pesseece ‘24. = _ By ye ee 
- We can add a second point to this “aggregation level’ or ‘object 


choice’ problem: Surely no_important country (US, USSR) or 
international group of countries (Europe: CERN) will close down, 


Or even put on ice, large prestigious uujects like high-energy physics 
centres, radio-astronomy centres, and.so on, on the basis of 
evaluation studies. These centres are simply too rare, too 


877°). 
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prestigios: At most, : national authorities will introduce. changes j in 
such a centre - ~forexample, a very obvicus intervention like closing 
down 4 particuize pat: “f the centre (a specific accelerator). Only 


when a relativcty arg. number of capensive, similar centres are 


present in OF = Jars -~ Fike the various high-energy physics 
centres in the 3.5 °°! ‘ate uINe : ma: ¢ itional policy lead to a 
gradual closing Gow: “if ax: Lat teetres. Sut in ali these cases 
participating groups of sues... ~ wi eircady have moved to other 
facilities. They wil! — agsin, +2 i casi in the West, where Scientists 
really can ‘change facii:tics’ -- predict” such a notional posicy 
measure. 


Some eee ‘Technical’ Problems 


face pee. egeegeee siesoge Co Ga Ygepebgeeeenioig!  § q@ecee. velco 


concentrating mainly « on IMs srecent ‘ ‘East and West’ paper. First. it 


is not true that mastering the sateen eget incitation Heil is 


manu We refer to our work.’ In manual scanning ran 
difficulties are. involved, not the least being a very obvious factor ake 


sophisticated search-software is much more recommicndable. 


Second, IM discuss and estimate three main sources of likely bias 


against Eastern bloc. accelerators.in. their citation data. We focus 


here on the first and the third: The first source of bias is that papers 


scar. ed by IM ‘lose’ citations from journals not covered by ISI. The 
third is that papers published in the main Soviet journals tend to 
contain less references, on average, than those in equivalent 
Western journals. IM do not explain why these sources would cause 


a bias particularly against Eastern bloc accclerators. In principle 


horh Eastern and Western papers scanned by IM. suffer from the 
citations that are ‘lost’ (first source) and from the shorter lists of 
references ‘in Russian journals (third source): _Why then should 


Eastern jearaals refer aint exclusively to other Eastern- ‘Papers 


agai evidence in support of this assumption. Inour opinion, an 


io) 
ERIC 


373 


546 Social Studies of Science 


Finally, 1M state that: 


The citation rate provides an indicator of the average impact per papet from cach 
ascelerator and thus allows for differences in the scale of research activity at cach 
facility. (EW, 338, fn 52; emphasis added) 


This is probably rior trie. In a recent study on characteristics of 


citation behaviour, we found evidence that suggests that the 


‘citations per publication’ is affected positively by the number of 


publications, and therefore docs not allow properly for differences in 


the scale of research activities.‘ 


Concluding Remarks 


We would like to. emphasize our appreciation of IM’s enthusiastic 


activities. The amount and the range of their work is impressive. 
Without doubt, IM have played, and still play, an important role in 


the development and application of quanti tative-empirical analyses 
of_ research performance. Their-work contains many. valuable 


elements. They have stated firmly that _oi= should not rely on 


bibliometric data in analysis of research performance: other types of 


indicators). They have clearly introduced the distinction between 
‘quality’ and ‘impact’. of reseazch performance — and, more 
especially, ‘impact’ as a theoretical concept, reflecteci more or less 
adequately in citation counts. In addition; they have defined clearly 


the concept of (partial) indicators. And we would finally mention, as 
a valtzble element in IM’s work, their effert —at least in thei: latest 
Papers — to analyze the causes of the observed differences in 
research output, impact nd peer-recognition. We firmly agree that, 
from the point of vie-s oi researchi pticy, one should obtain insight 


into the causes of the observed level of research performance in 
order to make proper, effective policy decisions, = = 
_-Nevertheless, our critique is evident. IM ‘promise’ too. much. We 
believe that IM’s method: of. ‘:onver;ing partial indicators’ can 
Provide at best a partial indication of contribution to scie:itific 


Progress, and nothing more tl:an that. Ii addition, performance 
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discrepancies should ‘lead to questions.addressed to these local 
peers, and to more detailed, gisalitative analysis. 


I. 5: Irvine and B: R: Mantin, “Basic. Research i in the East ana whe eet A 


Comparison of the Scientific Performance of High-Energy Physic:. 4.-.> <.siors’, 
Social Studies of Science, Vol. 15 (198%), 293-341 (referred to below as E W);: Irvine 
and Martin, ‘Assessing Basic Res-arch: the Case of the Isaac Newton Telescope. ibid. 
Vol. 13 (1983), sods ne 
2. B.R. Martin ar*. °.:/r “Assessing Basic Research: Some Partia! Indicators 
of Scientific Progress IF Soar + satronomy’. Research Policy. Vol. 12 (1982j, 61-90 


(RA). - 
3. H: F. Moed, W.. ae. ha werd G. 3. Frankfort andA. F. J. van Raan, ‘The Use 
of aonb Data Bo ote aerermcat of. digested Mb iclatelad _Remenh 


Research Policy Unii of the University o of Leiden, 1983), 1-19) | 

4. H. F. Moed, W. J. M. Burger, J. G. Frankfort and A. F. 1. Vin Raan, 
‘Caaracteristics of Self-Citation Behaviour: References to the Ocuvre of a Research 
Group’, in W. Callebaut-et-al. (eds), George Sarton Centennial (Ghent: 


Communication and Cognition; 1984), 472-73: 


Author's iddress: Research Policy and Science Studiés Unit; 
Stationsweg 46, University of Leiden, 2300 RA Leiden, The 
Netherlands. 
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Recwiices and Repti ies (esnitiniee 


The Case of the Disarpearing Caveat: 


A Critique of Irvine and Martin’s Methodology 


The work of Irvine. and Martin (IM). ina sisvided a_ wealth of 


statistical data to be taken into.account by riiuie policy makers and 


historians. Those analysts will interpret ine data in terms of the 


signific icance of what has. been counted. _This need. _for careful 


eset ied as the sensitivity shown in ‘the. peeenteersy 


introduction has been abandoned. The authors. have failed to avoid 
the question: What is research in science for? That is a black hole into 


which.-the.: analysis is irretrievably_sucked. First they impute 


unrealistic goalst to science, and then they us~‘ 1dicators which do not 


indicate levels of achievement of those hypothetical goals. 


_ This Response focuses ps their recent paperon ‘Basic Rescarchin 


the] East and West’ (EW). The object of that work is to evaluate the 
scientific outputs of high-energy-accelerators in the Eastern blo: 
compared to those ir the United States and Western. Europe. But ii 


one is to ‘evaluate’ science, or indeed anything else, performance 


must be matchcd to. gouls. When these are embedded in three such 


different cultural systems the issue ‘s particularly salient. 


_ The objectives of science are explored. by IM themselves in th: : 


recent book, Foresight in Science (FS).* Here it is explained itis thig 
conventional differentiation of ‘basic research’, ‘applied research’ 


and ‘experimental development’ may be useful for statistical 
purposes, but not.for science. policy (FS,2). Within ‘basic research’ 


one can distinguish in terms of the patron’s objcctives between’ pure 


or curiosity oriented research’ and ‘strateg =: =search’ (FS, 4). The 


Social Studies of Science (SAGE, London, Beverly Hills and New Delhi); Vol: 15 
(1985), 548-53 


38 


r) 
ERIC 


376 


Responses & Replies: Bud: Response to 1M 549 


former-is promoted for no ‘long term economic or Social benefits 
other than the advancement of knowledge’, whereas the latter is 
expected to ‘produce a broad base of knowledge likely to form the 
background to the solution of recognised current or future practical 
problems’ (FS, 4): The authors explain in more detail the character 
of strategic. research. This is defined, they emphasize, by the 


interests not of the scientists but of the sponsors, normally large 


science-based firms, or governments (FS, 5). The objectives are: a) 


toconiribute background knowledge required in the development of 
new technologies, and b) to develop links with relevant academic 
research communities which would enable the firm or government to 
exploit knowledge created through the effortsof others (FS,5). Both 
emphasize the importance of developing-understanding within the 


sponsored laboratory. Curiosity-oriented research is less carefully 
examined. It-is in their analysis an empirically residual class, basic 


assumes the quite different orientation towards knowledge for its 
own sake. It is admitted that the research ‘also leads to substantial 


educational benefits {in the form of highly trained scientific 


personnel), to various types of technological “spin-off” , and evento 
broader political benefits such as iticrcased. ational prestige and 


improved international cooperation,’ (EW, 299) but these benefits 
are written off as ‘secondary’ and trivial in an undocumented claim: 
“Yet these secondary reasons cannot explain why; over the last 
decade, nations have spent around $1000m a year on the subject’ 
(EW, 299). Nevertheless we learn later (EW, 301) of the belief that 


such research would contribute to national nuclear energy research 


during the . “50s and early 1960s: For the containment in time no 
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ev.dence beyond the rfictoric of scientists is produced. The very 
f°" >.7aragraph emphasizes the national prestige considerations 


lying behind the ast European effort in general, and the building of 
*he Serphukov reactor in particular. The emphasis is well justified by 


the survey of East European physicists relegated to footnote_73 


(EW, 340): Although we are left in the dark about Western 


decisions, from Greenberg and others we know that these have also 
been politically complex. Gibbons’ study of the 300 GeV accelerator 


showed the kind of considerations underlying policy advice to a 


community based: That is clear from the initial explanations of the 
chosen indicators. But the author's Self-consciousness about the 


scientific ‘production’ and to ‘progress’. Neither can be :~easure-'; 
instead their-magnitude is to be judged through the use of Dartial 
indicators. These are, we are specialiy reminded, ‘mental constructs 
not to be confused with reality’ (RA , 66, fn 11}. Three are chozer in 
the radio-astronomy paper, and are used again in the study of high- 
energ;' physics: numbers of papers, citations, and peer asses:irent: 


directly. It is recognized that a complicated series of intellectual and 
social processes is invoived in peer review. What determines 
scientists’ opinions of their colleagues is recognized to be most 
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(RA, 70). Itis only with the last that the authors are concerned. They 


claim no necessary relationship with either qualit’ or importance. 
An elegant implication of this approach is that it taxes into account 


all those social factors beyond some notional ‘importance’ of the 


paper. These may include, we are informed, the location of the 


author, and the prestige, language and availability, of the publishing 
journal. It is not an absolute factor, varying according to the 
“cognitive and social location of the assessor’ (RA, 70-71). Finally, 
the authors claim _that_number_of papers can be used to.indicate 


contribution to ‘scientific progress’. This is done with caution since 
we.are told that ‘we have to consider why scientists publish papers, 
esent valuable results; but 


and to reatise that they do so not onl: to 
also.:or social, political and career rea. ::5," (RA; 65). _- 


- These indicetors, it is claimed, can be used toget‘ier to compare 
the contributions to ‘scientific piogress’ of laboratories. Given their 
culturally sophisti-ated concept of ‘progress’, it is important that the 


laboratories coripared operate within the same culture. As IM say: 


What is ideally required is that there should be two or more groups, working in the 
same specialty over a similar time period, publishing in the same journals, 
supported -with a roughly similar level of resources; and situated in a similar 
institutional context; (RA, 75) 


The moral underlying this methodctogical account is that statistical 


results have to be interpreted before convergence or otherwise can 
be concluded. Not only do raw data need to be analyzed befure being 
turned into conclusions about ihe magnitude of the indicators: those 


indicators themselves need to be interpreted. Each indicate a 
different phenomenon. Only waen the interpretations coincide can 


one properly conclude tiiat they have ‘converged’, and on what they 
have converged. mone S035 Bossiele eo ees ee 2 oe 
_ In IM's ‘East and West’ study of high-energy physics, the care 


taken with the collection of data is quite remarkable. Perhaps only 
those_with experience of such work can appreciate the assiduity 
represented in a few short tables. The following conclusions are 
reached: a) fewer papers were published based on work in Soviet 
reactors; b) these papers were less cited_in_eleven major 
international journals taken together; c) anonymous physicists 
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perfunctory. Conclusions ar are drawn only about overall magnitudes. 
The understanding of what may be happening is derived from the 
interviews, with all their problems. Descriptions of East European 
weaknesses are combined with explanations of them. All depend on 
the complex, and uninterpreted, interests of the commentztors. We 
can only note, without understanding the implications, that. the 
number of Eastern European scientists interviewed was aii order of 


magnitude smalter than that of the Westerners; and that we are_not 


told of the method by which the interviewees were chosen..This is 
particularly problematic, given the small number of East 
Europeans. 

- The analyses of publication counts sided citations could have beet 
richly suggestive, given the author's sophisticated understanding of 
publication and citation as social processes.. However, the_only 


meses interpretation of Soviet_publicaticn counts is that one 


might expect that with iess pressure to publish trivia the average 
impact would be higher. Citation data, despite its complex 
significance, is subjected to similarly slight analysis.. As IM had 
emphasized, it indicates ‘impact’, which is determined by the 
relationship between producer and reception -communities, 
mediated by language dnd a whole host of social and cultural 
phenomena. oe a Peep at the implications is Babee = by. the 


eae eae ‘papers from the Soviet. reactors sendy in the 
West, and those published in local journals. This is explained merely 
in terms of objective ‘quality’. Meanwnile the segregation of Eastern 
and-- Western. scientific communities: is not examined and its 
implications lie unexplored. The authors are also silent on_the 


intellectual and social objectives of publication and citation in East 


and West. Instead there is a silent shift from the category of ‘impact’ 


to that of quality. There. is reference_to the poor ‘record’ of 
Serpukhov. The result is that although the authors claim to have 
found the convergence of three separate indicators, what they have 
is one detailed indicator. (the interviews) whose results give some 
meaning to the statistical-analyses ot citations and production. We 
are not provided with sei ee access ta the meaning of those 
tables: _ 

_. The interpretive problem: with all the indicators is the same: What 
are their implications for our understanding of the relationship 
between the scientific communities? Unfortunately, since the 
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worldwide, the accumulation of local expertise and the mechanism 
of its interchange are defined as irrelevant. The imputed objectives 
of scientific activity rule. out realistic interpretations of the chosen 
imputed objectives and the uninterpreted indicators—are therefore 
interrelated... - - 

_The historical account is consequently bedevilled by! two empirical 
problems: it is not clear what phenomena are indicated here; and we 
do not know what phenomena we, as evaluators, should _be 


interested in. In view of IM’s meaning of ‘outputs’, their conclusion 


that ‘the scientific outputs from each Eastern-bloc accelerator have 
been small. in comparison with the nearest equivalent Western 
facilities’ (EW, 334), may be trivial. 


1. Ben R. Martin and John Irvine,--Assessing Basic Research: Some Partial 
Indicators of Scientific Progress in Radio Astronomy". : Resear? salad i Vol. 12 
(1983). 61-90: : 

2. John Irvine and Ben R. Martin, “Basic Research i in the East and. West: ; A 
Comparison of the Scientific Performance of High- -Energy Physics Accelerators’, 
Social Stadies of Science, Vol. -15 (1985), 293-341. - - 

-- 3.-John Irvine-anc Ben_-R. Martin, Foresight i in Science: Picking the Winners 
(London: Frances P’>ter, 1984), esp: 2-7. 

_ 4 Michael Gibi: aus, ‘The CERN 300 Gev Accelerator: A Case Study i in the 
Application of che Wieabers CHIEHE Minerva, Vol. 8 (1970), 180-91; ;on American 


(Harmondsworth, Middx: Penguin, 1969), 2% 1-302. 


Author's address; The Science Museum, South Kensington, 
London SW7 2DD, UK. 
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The Possibilities of Science Policy 


Irvine and Martin (IM) have been responsible for some scrious 
questioning of British science policy. To-date, the outcome of their 
work seems likely to be beneficial for those who seek some basic 
nourishment from the research budget, rather than the five-course 
dinners of Big Science. Nevertheless, their research is not. without 
flaws, norisit obviously the best way for science policy to make use of 
recent work in science studies: I will criticize it on two grounds. First, 
it-misses the cognitive ‘wood’ for the institutional ‘trees’, because of 
the way in which it disaggregates science. Science policy ought not to 
be about maintaining efficient institutions, but about maintaining 
the sort of cognitive. community that will produce the desired 


scientific products.' The second, interrelated, criticism is that their 


measures of scientific output are inappropriate, since they reflect 


only the internal reward system of science; optimum performance 


according to such measures is not necessarily optimum for science 
policy. My criticisms of their work on.the particular topic of high- 
energy. physics institutions? are implicit in these more general 
remarks. eS ete a eee ee See eeee tees 
_ IM have studied scierice by breaking it up into units of comparison 
defined by what I will call ‘non-cognitive boundaries’. Usually the 
boundaries chosen have been those of institutions — the laboratory, 
the university department, the discipline, the nation. The analysis 
has then turned on.the comparison of. output/input ratios of these 
units. The greater the output/input ratio the better — that is, more 
cost-effective — the unit. This is certainly an improvement _on 


goals in view, it is essential to start by disaggregating science 


Social Studies of Science (SAGE, London, Beverly Hills and New Delhi), Vol: 15 
(1985), 554-58 
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accordin ng to cognitive Hanes than institiitional tscuindaries = thatis, 
to think of science as being made up of sets of research areas which 
involve sciertists who interact, or mutually refer, acrossinstitutional 


boundaries, because_of their common cognitive. interests... The 


boundaries of such areas do not necessarily map on to the boundaries 
ofi ‘institutions. eetaee 


Turning to IM’ 's measures of output _ publication counts; citation 
statistics and peer-evaluation— these ali reflect the reward system of 
science. This is probably why these ‘converging partial indicators’, 
converge so readily. But the reward system of-science has an 
‘atomistic’. ethos. Prizes, citations, peer appreciation, and. the 
accolade of having made a major contribution, do not go to all those 


who take part in a Scientific. endeavour, but. to those who complete 


the work first. As Medawar says, ‘Much of a scientist's pride. and 


sense of accomplishment turns .. . upon being the fi irst to ) do 


reward System does map on to institutions. That is why IM’s 


measures. of output z appear to make sense. Institutional units can be 


readily compared in terms of the aggregate rewards their members 


receive. The result says little about the overall desirability or 


usefulness of the research, but much about priority of knowledge 
claims. 


A Fable 


We can explore the difference between foreseeable policies basedon 


cognitive criteria with a thought experiment —or, 1 more : properly, a 


unrealistic, assumptions about how science works; one : of these is 
that the new discoveries of Big Physics are overdetermined, and 
another is that most top level scientists are fairly competent. Events 
in the fable are, of course; somewhat more neatly organized than 


they are in real life, but the principle still stands, I believe. ___ 


Imagine two laboratories — ‘Paragon’ cost. $10M (Megabucks) 


per year to run, while ‘Querulous’ costs $9M per year. In.a typical 


year Paragon produces ten major findings while Querulous produces 
none, merely rediscovering the tindings of Paragon about six months 
later. Paragon fvbdlishes many important papers, its scientists win 
honours of various sorts, and it is highly esteemed in the scientific 
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Paragon. ‘Taking the two laboratories as separate-and individual 


entities we might conclude that to close down Querulous would save 


$9M per-year for little loss in-terms of discoveries in physics. But, if 
one thinks in terms of what will be discovered under various policies, 
instead of who has been most_rewarded, and if one sets aside 
considerations of equity, it turns out that it could be more sensible to 
close Paragon! - . ~~ 
__This_is how such an_apparently_ counter-commonsensical 
conclusion can be reached. If we closed Paragon we could save not 
$9M but $10M per year. However, perhaps all the same discoveries 
would still be rade, but it would simply all happen six months later: 
The same publication outlets would still need to be filled and, since 
the reward systzm would not change, the same prizes would be given 
away, but it would now be the scientist from Querulous who had the 
priority: they would be the recipients of their colleagues’ esteem, 


be just as many prize winners as before.° (Actually, the metaphor fs 
rather harsh, for Querulous’s scientists are not. dumb, just slightly 
slower than Paragon's, It is the reward system itself that makes 
tardiness look like incompetence.) Under these circumstances we 
would have the same science, a bit later, for an extra saving of $1M 
per year compared with the ‘close Querulous’ scheme. Under these 
assumptions it is the absolute cost of running the laboratories that 


that treating science as an aggregate of individual laboratories, or an 


of purely cognitive goals — producing scientific findings — the 
institutional/atomistic approach can _ generate. _ irrelevant 
impressions. Of course, the fable can give rise to no real policies, 
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year | to run n before twelve months had elapsed. The. point re remains, 


however, that different ways of approaching thc measurement of 
scientific quality can lead to widely different conclusions. 


Coccioion 


substantial progress in studies that break up Science a reference to 


cognitive, rather. than institutional, criteria. The-sociology and 


history of scientific knowledge have developed methodologies for 
analszing passages of -scientific activity, irrespective of their 
institutional locations. It is. time. that these sttidies Started. to be 


to investigate the balance of the different sphasea ols science’ ‘(normal 


and extraordinary) and the conditions of_their_survival;’ the 
conditions. for the maintenance and transmission_of bodies of 
scientific skill; the way that the allocation of research funds affects 
the practice and findings of scientific research; and the way in which 
the scientific community constrains, or encourages. the production 
of novelty. While quantification of these-exercises will be difficult, 
we should not allow our attention to be distracted from the serious 
problems of number fetishism —-as John Irvine must know." Were 
these important goals to_be pursued with IM’s energy, we might 


develop a science policy that would do_more than offer short-term 
solutions to perceived cost-ineffectiveness. 


© NOTES 


This is a a modifie ed version nof part. ofa paper first precited at the Scince Studies 


Committee meeting at Imperial College. London. on 25 June ‘1983. Another = of 


53. I am grateful t to ) Jay Gershuny and John Cullis for very helpfut comments onan 


earlicr mate of this Response. 


_1.-AsIM soint out; there mapbe® non-cognitive’ aims forscience policy, ¢ too: 0: For 


example, economic and technological spin-offs from scientific activity might be seen 
as more important-than the science itself; scientific training is vital, and prestige and 


political c considerations are not to be ignored. However, a policy which was directed 
n izing some orall of these goals would not use IM’s methods, either. - 
__2._J livin and B: Martin, ‘Basic Research in the East and West: A Comparison 
ar High-Energy Physics Accelerators’, Social Studies of Science, Vol. 15 (1985), 293— 
341. 
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Martin and Isvine, ‘An Evaluation of the Research Performance of Electron High- 
Encrgy Physics Accelerators’, Minerva, Vol. 14 (1981), 408-32; Martin and Irvine; 


4. A general criticism of ‘methodological triangulation’ is that it is not clear what 


one does when the measures do not converge. 1 agrec with Krige and Pestre that non- 
convergence is the more interesting and demanding case. As will be seen below; Ido 
not agree that IM’s measures converge on scientific quality in any straighttorward 


way, since priority determines rewards to too great an extent. See J. Krige and D. 
Pestre. ‘A Critique of Izvine and Martin's Methodology for Evaluating Big Science’. 
Social Studies of Science , Vol. 15 (1985), 525-39. : 

5. P. Medawar, The Art ofthe Soluble (London; Penguin, (969). 96. -_ 

6. Some economists would say that scientific rewards are a ‘positional good’. 


__ 7, See H. M. Collins, Changing Order: Replication and Induction in Scientific 
Practice (London and Beverly-Hills, Calif.: Sage, 1985). - = - = st 

_. 8. See J. Irvine, I. Miles and J. Evans, Demystifying Social Statistics (London: 
Pliito Press, 1979). 


Author's address: Science Studies Centre, School of 
Humanities and Social Sciences, University of Bath, Claverton 
Down, Bath BA2 7AY, UK. 


Responses and Replies (continued) 


Evaluating the Evaluators: 


A Reply to Our Critics 


tm 


Ben R. Martin and John Irvine’ 


differences we_have_with our critics, it is perhaps first worth 
identifying the issues on which we are in agreement. Four stand out 


Social Studies of Science (SAGE, London, Beverly Hills and New Delhi), Vol. 15 
(1985), 558-75 
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a) research addressing science policy i issues is jeaicently needed: 
_-(2) advances in the area of science studies over the last two 


decades. mean that it is now possible to tackle substantive science 


policy issues ita moresystematicmannez; — --.-§ _. ___.___ 
(3) bibliometric studies have begun - to focus. on ‘issues. of 
importance’ and ‘have made a marked impact',” even succeeding in 
certain cases in providing ‘a wealth of statistical data to be zaken into 
account by future policy makers’;> - 
(4) our approach i is certainly capable of beinigimnpraved upon. 
In view of the: unanimity 9n these points, we are disappointed at how 


little our critics have to_ fer by way of concrete suggestions as tohow 


one might improve the techniques_ we have been attempting to 
develop to provide systematic and reliable information for science 
policymaking. . -Let us consider in.turn the points made in the four 
critiques and the improvements (if any) in evaluation methods and 


approaches that they suggest. 


Policy Research or Fairy Tales? 


Harry Collins begins. by congratulating a us for provoking ‘s ‘some 
serious questioning of British science policy’. ‘Nevertheless’, he 
continues, ‘their research i is not without flaws, nor is it obviously the 
best-way for science policy to make use of recent work in science 
studies’ .* What are these ‘flaws’? 

_ The first is that we have taken as our unit of analysis research 


institutions_rather than_units. reflecting cognitive boundaries.° 


According to Collins, policy research should ‘start by: disaggregating 


science according to cognitive rather than institutional boundaries’.° 


This is a fundamental difference, and one on which we. disagree 
strongly with Collins. In basic science, at least, policymakers often 
take decisions on whether to fund research groups, departments, 
laboratories,. and. Centres, as opposed to entities defi ned- ‘purely by 


of analysis for the. study of. certain questions i in the sociology of 


science, for science. policy research. the. former is important and 


clearly cannot be ignored. Examination of the expenditure of most 
national research-funding agencies shows that research groups, 
laboratories, and so on — that is, institutionally defined entities — 
account for a large proportion of their total budgets (some grants 
are, of course, made to individuals — most importantly to open up 
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new areas — but these. -account for only a fracann of the funds): 
There are, of course, other cases, generally involving more applied 


research, where such Organizations may have to decide whether or 


not to support a field (and in our more recent work we have tried to 


address this type of question’). However, even then, once sn agency 


has decided tosupporta field for strategic or other reasons, it still has 


to. choose which research groups or institutions to support, so 


information relating to their past and likely future performance is 
just as vitat: In short, Collins.is wrong to assume that what is most 
interesting from the point of view of a particular sociclogy of science 
sanelpeanit Pomicides exactly. with ‘the. main. apes of palicy- 


maintaining efficient institutions as well as ; maintaining. the sort of 


cognitive community that will produce the desired. scientific 


products’: (As an aside, we would also point out-that much 
interesting sociology of science would be lost if Collins’ demarcation 
were to be universally adopted. ) _ Bees oe 

The second criticism is that the indicators we have employed 


reflect the_atomistic reward system _of science, with all the prizes 


going to those who are. first..In_fact, only one of our indicators 


(hig; *y-cited papers) relates directly to ‘discoveries’ or ‘coming first’ 


— the others relate much more to taking ‘part in 4 scientific 


endcavour’." For example, our study of world high-energy physics 
showed that between 1961 and 1982 there were very few occasions on 
which CERN was ‘first’ (nearly all the -crucial-discoveries and 
advances over this. period were made _in the United States).’ 
However, the other indicators suggest. that CERN accelerators were 


perhaps the most_successful in terms of experiments yiclding more 


precise measurements and better statistics, a fi indie in line with the 
results obtained from 180 interviews. "" 

Collins then attempts to demonstrate byi means of* aiairy tale’ ‘the 
difference between foreseeable policies based on the reward system 
and institutional units, and those based on cognitive criteria’.'' The 


tale, we are informed, ‘rests on some substantial, but_not_totally 


unrealistic, assumptions about how science works’. Onc assumption 


(apparently - unrecognized -by Collins) is that the international 


dimension of science can be completely ignored. (This assumption 
may actually hold in Collins's own research area, but that is 
something of an exception.) Besides Paragon and Querulous in 
country (or continent) A, there wou!d be other Paragons and 
Querulouses around the world. Hence, if a Britisa Research 
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Council, for caailipke, were to decide to close the world leader i ina 
particular field, the probable result would not be that the same 
discoveries would be made six months tater by a less costly UK 
group, but by Texas Megabuck Lab or Nippen Lab. Furthermore, 


Collins’s fairy taie, even ifit had been based on realistic assumptions, 


entirely misses the. point. In most cases, those. outside. the field 


concerned will not know that Paragon has been.making ten major 
advarices ayear\ while Querulous has been making nc none. - Scientists 


differerice between the two, but, for reasons related partly to the 
existence of Vested interest groups, they-may not reveal the true 
extent of the difference to others. Norare they likely to have as wide- 


ranging an understanding of the factors structuring research 


performance as that, for example, yielded in our study of high- 
energy. physics accelerators. We have seen our task as one. of 
providing systematic information on research performance and the 
factors structuring it,-in a form accessible not just to researchers in 
the field. concerned. -but also. to Policymakers, Scientists in other 


sraraparency in in the scientific decision making process. eee 


_ Having argued that our work is not ‘the best way for scierce policy 
to make use of recent work i in science studies"? (and employed his 
measurement of -scientific uae can-lead to widely different 
conclusions’), what does Collins suggest might constitute ‘the best 
wy’ forward? As his first example, he proposes that ‘we could begin 
to_investigate the balance of the different “phases of science” 


(normal and extraordinary) and the conditions of their survival’.!" 


But does this (or indeed any of the other examples Collins cites) 


actually correspond to the primary concerns of policymakers? Over 
the last seven years, we have devoted considerable effort to trying to 
establish just what are their'main i interests and problems. Almost all 
the many policymakers in-the UK and overseas who have discussed 
this with us have stressed the néed for better information on the 
performance of research groups, departments, facilities and 
laboratories, and on where their country stands in international 
terms in particular fields.'* As far as we can recollect, not cne has 


mentioned any of the examples listed by Collins. ' 


394 


r) 
ERIC 


562 Social Studies of Science 
The Case of the Misread Papers 


Iti is s difficult to know how to take thea criticisms by Robert Bud when 


ceaperinicata Eos _is penis inadequate. In 
particular, within basic research one can now identify two very 


different types_of research activity, namely ‘curiosity-oriented 


research’ and ‘strategic research’ — the latter being work where no 
specific_end-product or process can yet be identified (so it is not 
‘applied’ research) but where the research is expected to produce a 
broad base of knowledge likely to form the background to the 


solution of practical problems. The book then concentrates on 


methods for looking at the longer-term future for strategic research. 


It does not, as Bud claiins, dismiss curiosity-oriented research as an 


empirically residual class, but rather argues that its greater intrinsic 


unpredictability. makes attempts at longer-term forecasts less 
reliable and therefore not. so worthwhile. As with any 
categorization, ‘curiosity-oriented research’ and ‘strategic research’ 
are to some extent_ideal types, but it is not difficult to think of 
examples of research activities which are predominantly one or the 


other. Most basic research biotechnology, for example, is primarily 


‘strategic’, while_at the other end of the spectrum we would place 


almost all astronomy and high-energy physics. We can only assume 


that Bud’s view of high-energy physics as strategic research is linked 
to his. belief.(stated in two places) that experimental high-energy 
physics is carried out on ‘reactors’”’ and therefore is presumably in 


some way part of nuclear. « energy research. In fact, the. two fields 


parted company in-the 1950s, and the links between them (both 
cognitive and social) are now almost _non-existent_(or_at least_no 


greater. than with other fields of physical science). This is the reason 


why in our: study we concentrated on evaluating the performance of 


accelerators in-scientific. terms. We would, however, welcome 
positive suggestions on how to extend the evaluation to cover 
‘extrinsic’ goals.'® 

__ Putting aside the doubis engendered by such a mistake, what can 


insufficient attention to examining the goals of scientific research 
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(although Bud doesnot actually tellu us what he thinks these are); and 
that we have used indicators ‘which do not indicate levels of 
achievement of those hypothetical {sic} goals’."” Right from the start 
of our research programme, we have explicitly recognized that one 
cannot _use the same indicators to evaluate different types of 


research. If Bud were to read, for example, the report on a study of 


mechanical, electrical and electronics engineering that we carried 


out for a 1981 Norwegian Royal Commission, he would see that.a 
very different approach and set of indicators was adopted to evaluate 
these more applied areas, whose objectives clearly differ from those , 
say, in high-energy physics. (We laid great stress, for example, on 

‘customer review’ —that is, obtaining the views of industrial firmson 
the research activities being assessed —and produced a large volume 
of quantitative data relating to this.) ___ 


__A second criticism relates to. our. comparison. of the research 


performance of accelerators in the Eastern bloc, the United States 


and Western Europe, with Bud pointing out that the accelerators 
‘are embedded in three such different cultural systems’.?! Whereas 
one response to these cultural differences may be to throw up one’s 
arms in horror and Hise that no valid ee can. seal 


rel ied to the cultural. differences: (Eastern. and Western high- 


energy physicists are not in fact the two completely distinct and non- 


interacting communities that Bud implies: They attend many of the 


same conferences, use each other's accelerators to a limited extent, 
and publish in essentially the same body of international journals, 
with East Europeans having made i inereasing use of West European 
journals over the last twenty years.) 

_Next, Bud chides us that ‘the interpretation of the. quantitative 


data is. perfunctory: Conclusions are drawn only about overall 


magnitudes’ .” As with any piece of research (but particularly one in 


anew area). the analysis could have been taken further. We have not 


attempted to .solve.all the problems in making East-West 
comparisons of scientific serformance at once and have also been 
careful to avoid drawing over-ambitious conclusions (that is to say, 
the caveat has not ‘disappeared® as -Bud-claims) which cannot be 


adequately supported by data with all the limitations we describe - If 


Bud has specific suggestions to. make in relation to interpreting the 


empirical data further, we would appreciate the opportunity to 
consider them. 
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Overall, Bud, like Collins, seems to exhibit a myopic tendency to 


see everything from the perspective of science studies rather than 


science policy. His ‘problem with all the indicators is the same: What 


are their implications_for_our_ understanding of the relationship 


between the scientific communities?" We would merely note that 


this is not a problem that science policymakers have identified to us 


as onc of their principal concerns. 


The CERN Critique 


Thistated aim ofthe« critique by: John Krige and Dominique Pestrei is 


simple and admirable — it is to defend ‘intellectual rigour’ .*" It is 
therefore alarming to find them beginning the presentation of their 
case by quoting what somcone stated at a seminar had been said to 
him on some other unspecified occasion by ‘a Chinese colleague’ 
who was in turn ‘quoting’ two other people. (Did Marx and Engels 


actually both Say. -anything as_simplistic_as that — - we would be 


Response, this unsubstantiated anecdote has been elevated to the 
status of evidence which ‘makes the point’. Is this more ‘rigorous’ 
than our structured interviews with over 180 high-energy passes 
each of whom then completed a detailed attitude survey? _ 

_ After presenting a clear summary of our work on big science, 
Krige: and Pestre commence their critique by arguing that such non- 


scientific benefits as ‘manpower "training, technological s spin-off and 


energy physics. This would seem to suggest that they see such 
benefits as ‘primary’ — that is, of equal (or greater?) importance 
than the scientific question of how much the research will increase 
our knowledge of the matezial world. (Krige and Pestre do actually 


contradict this two. ve rates ‘We 3 aes not. claiming. that Sa 


Saas are they? A. little more rigour is el here. eyT This t raises 
the interesting (and unanswered). question, ‘How ‘poor would the 
scientific. performance of CERN have to become before scientific 
criteria finally over-rode non-scientific ones?’ Or are we to agree 
with Moed and van Raan that centres like CERN ‘are simply too 
rare, too prestigious’ ever to close down?” 
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“In concluding our. analysis of CERN’ Ss past performance. we 
discussed the political, educational and technological bencfits it has 
yielded.”” We. are, however, less. convinced than our CERN 
colleagues by the findings of Schmicd on technological spin-off from 
CERN.” While it may be comforting to CERN to have -a- study 


claiming to find that the industrial contracts placed by CERN have 


resulted in increased ‘economic utility’ several times greater than the 


Original value of those contracts, this should not blind us to is 


methodological inadequacy: First. of all, the concept of ‘economic 


utility’ employed in the study would be. unrecognizable to most 


économists. Secondly, the methodological approach of asking firms 
to. Se ego contracts | that have drawn - sae work 


there is the question a how. sriach reliability ci one.can place on the 


answers of firms dependentt toi a 1 greater or lessere extenton CERN fe for 


Lastly, the opportunity costs are completely ignored —‘is the level 
ae sae pues ean hat it would fave been. if the 


confidence in the results of oto than do Kies and | Pestre. 


energy physics i in in general, are supported more for the contributions 


they.are expected to make to scientific: -knowledge.’! In our paperson 


CERN,” we quote various explicit statements from CERN to this 


effect.>*. Furthermore, few of the scientists involved attempt to 
defend the funding for their work by reference to ‘extrinsic’ benefits: 
our attitude survey revealed that twice as many felt that expenditure 
on the field could be justified only in scientific terms as believed it 


could in terms of the spin-off it generates. Krige and_Pestre’s 


observation that Big Sciences like high-energy physics *have a kind 
of role analogous to that of military research in the economical field’ 
is perhaps more revealing than. they intend, given. the. extensive 


literature demonstrating how military R&D, while it does yield 
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occasional important spin-offs, at the same time locks up scarce 


R&D resources that might more profitably be employed in other 
forms of research.™ ee aoe oe ees oe ta ee, ye eS, ee 
The next point made by Krige and Pestre is that we ‘rely essentially 


on the average feelings of the scientific community, leading us to 


doubt whether they can produce assessments substantially different 
from those obtained through a more classical panel system’.*> Here, 
it is important to distinguish. between conventional peer-review 
(involving a small number of referees or experts’ on apanel) and our 
extensive peer-evaluations drawing in very large numbers_ of 


researchers. across. different countries and based_on. structured 
confidential interviews and attitude surveys. Because the latter 
approach yields relatively consistent results, it does-not logically 


follow that conventional peer-review is ‘rather reliable’.* Indeed, 
we have encountered several instances where the-existence of an 
‘oligopolistic’ -situation in a research field has led to the two 


approaches yielding very different. results: One of the most 


United States.” where conventional. peer-review continued to 
Suggest that this was the top priority for US high-energy physics long 


after most researchers had privately recognized that this was no 
longer true (and admitted as much to us in interviews). By the time 
the project was finally aborted, some $200m had been spent. : 
‘Another important problem with conventional peer-review is the 
inherent lack -of accountability to those outside the specialty 


physicists, for example, to conclude that_existing accelerators are 
successful and that a proposed new facility is an absolute priority, 


bibliometric data on research performance, or external assessments 
of the likely future performance of new facilities. If presented in a 
form accessible to those outside the field, such information can play a 
major role in providing evidence to other scientists competing for 
scarce funds (as well as to policymakers and the general public) that 


decisions are well founded. -In. this way,-one can begin to achieve 
more open and transparent decision-making in basic science G eahe 
_ Krige and Pestre, in fact, provide. a good example where such 


accountability may have been somewhat lacking. At the time it was 


built, the CERN SPS accelerator was criticized by some, particularly 


Q 
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in the United States, for being over-engineered and ‘gold-plated’ (it 


‘gold-plating’ have there been which have nat paidoffin thisway? At 
what cost?. And were CERN member states. aware, when. they 


rigour, our critics must recognize that one example where ‘gold- 
plating’ did pay off by no means proves that this policy always 


constitutes the best use oflimited resources. ==. = = 
_ As for the ‘rhetorical rules’ identified by Krige and Pestre, we take 
these as not wholly uncomplimentary. We would, for example, 


The Leiden Alternative? 


This brings us to the criticisms of Henk Moed.and Anthony van 
Raan. The first relates to the various indicators. that we have 
employed and the extent to which they converge. The indicators, 


although related to some extent, do nevertheless reflect slightly 
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indication of the overall scientific production of a research group, 
while numbers_of_ papers_per researcher or per dollar reveal 
something about its productivity. The average number of citations 
per paper is a measure of the impact those publications have on the 


scientific community, while peer-rankings (where scientists rank the 
contributions of different institutions) provide evidence. on the 


perceived significance of the results, Lastly. data on the distribution 
of highly-cited papers reveal which groups have been responsible for 
the few key ‘discoveries’ or advances in a specialty, while aggregate 
citation statistics reflect the overall impact of a large number of 
incremental additions to knowledge. (Contrary to what some critics 
suggest, we do not claim that any of the indicators measure the 
‘quality’ of research.) As we have spelt out in detail.*' all these 
indicators are ‘partial’ in nature. However, in the real world of 


The starting point-when we began work-on issearch evaluation 


convergent results. This they did in the case of electron high-energy 
physics, radio astronomy and optical astronomy. In the latter, for 


converge in this instance. As we stated when reporting the very first 
research evaluation results, the fact that the indicators converge ina 
given case does not ‘prove’ that the results are 100 percent certain — 


the indicators may all be ‘wrong’ together.“ However, if a research 
facility like the Lick 3-metre telescope produces a comparatively 
large publication output. at fairly low cost, if.those papers are 
relatively highly cited, if it yields many of the highly-cited papers.in 


same finding was arrived at by a panel of three or four ‘experts’ 


total (even though each paper on average has a reasonable citation- 


ped 


49 


re) 
ERIC 


Ri 


Responses & Replies: Irvine: & Martin: Reply to Critics 569 


: _TABL E i 
Output Indicators for Oplicat’ Telescopes — A Suemimary® 


Lick - KPNO CTIO - INT. 
3-metre 2:1-metre 1.S-metre  2:5-metre 


Average ho. of ” __ - 

Papers p.a.. 42 43 35 7 
1969-78 

Cost per paper 2 = = Ss 
in 1978 £13k £7k £6k £63k 
tions to work oe a 23 2 
of past 4 years 920 70 580 140 
in 1978 


Av. citations 

per puper 4.2 3.3 3.3 3.6 
in 1978 

No. of Puperscited - 

1lor More times 4 31 21 4 
ina year. 1969-78 


ranked (oman ‘the bottom of a list of 12. fale by 50 


astronomers, we would be reasonably confident that its scientific 


performance had not been particularly good in world terms. Even 
so, we have always stressed that such findings need to be interpreted 
with care , and that rather than being used to replace the peer-review 
process, they should be fed into it to enhance its effectiveness and 
help keep decision-making ‘honest’. 

_ As for the concept of ‘convergence’, like any. notion when ft rst 


formulated it may initially have been somewhat ‘poorly developed’. 


However, as more empirical studies have been completed, so the 
conditions under which the indicators might be expected to converge 


have become clearer. First, the less satisfactorily research groups are 
‘matched’, the less convergence is to be expected. The convergence 
forthe four radio. astronomy. centres was reasonably -good. — 


certainly much better than when we compared CERN (which then 


had three major machines) with other high-energy _ physics 


laboratories (mostly operating just one accelerator each). Secondly, 


in a period of revolutionary change within a field, the facility 
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producing the most-highly-cited papers and ‘discoveries’ (as SLAC 
did in experimental high-energy physics during the mid-1970s) is 
likely to be judged the most successful even if it is not the. world 
leader in terms of numbers of publications and citations. Thirdly, 
convergence may be expected if the indicators are applied to 
research groups working in an internationally homogeneous field, 
but not if the field consists instead of several distinct , non-interacting 


communities (which do not attend the same conferences, publish in 
the same journals, cite each other's work, andsoon).** For situations 
between these extremes, where there is some interaction but not 
complete international homogeneity (as we encountered when 
comparing high-energy physics in the East and West“), one may, for 
example, first have to adjust the indicators for differences in 


publication and_referencing practices. This is, however, an area 
where more work is needed. - -- - ee ae 
- Another disagreement we have with our Dutch critics concerns 
the respective merits. of computerized and manual-scanning 
approaches to research evaluation. We have no objection to 
computing per se, and indeed use outside data-bases where there are 
significant advantages — for example, the NSF Science Literature 
Indicators Data-Base, which we have used todraw conclusions about 
the overall scientific performance of countries across research 
fields:*” However, there are certain tasks-in research evaluation 
where a manual scanning approach is the only option: 


(1) -In some cases, compu -ized scanning is too expensive because of the high 
access costs to data-bases (research evaluations should not cost more than a 


fraction of the research-being evaluated). ee 
(2) There can. be difficulties with using existing computerized data-bases in 


defining the boundaries of fields.” In the case of the CERN study, for example, it 
was necessary to construct our own data-base on experimental high-energy 
physics. Here, there. was no option to a manual-scanning approach because of the 


need to read large numbers of physics papers in order to establish which related to 
experimental high-energy physics and which did not._ . 
(3) Without reading the papers, it is generally. impossible to establish which 


research facility has been used to produce the experimental results. (this. is 


necessary for carrying out analyses of institutional and/or national performance). 
(4) Similarly, reading is essential to distinguish between-experimental and 


theoretical papers in a given field, something which is vital from a policy point of 
view for areas where the costs of experimental and theoretical work are very 
different. 


it is therefore misleading to pretend that the choice between manual 
and computerized scanning ‘all depends on the search-algorithm’.” 
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For the tasks mentioned above; the best search- software i is still tobe 


found in the human brain: Furthermore, the worry. about ‘the 
scanner getting tired (or even crazy)"*' needs to be counterbalanced 
by the benefits associated with actually reading the papers and 
becoming immersed in their content. 

-Since Moed and van Raan hold-up their own work as an exemplar 
of bibliometric assessment, it should. be pointed out that their 
approach is in our view somewhat flawed in that- performance 


indicators are applied to different departments within a single 


ai akers that i is, nes are used to. draw. comparisons between 


bibliometric data tse such groups ae peer computer print-outs 
from commercial data- banks), we harbour Las eee S about 


results. - : ee = 


Moed and van Raan’ s fi final criticism is that ‘performance analyses 
of large facilities such as accelerators have a limited relevance for 
research. policy’.*? In their view, ‘as soon as, at least in the West, a 
specific facility — like an accelerator : —~ its position at ae 


front”, scientists will move to other facilities. . ott would. . be 


is misplaced for two reasons. First, it assumes. 2. peed labour 


market within science completely at odds with the actual situation 


(even in the West).** Secondly, central facilities account for a large 
proportion . of the-- expenditure by _ national research- funding 
agencies,” and for Big Sciences like high-energy physics the main 


policy. decisions focus. on. whether to fund a centre or a new 


accelerator or detector, not on which university user groups to 
support. 


Let_us conclude as we began by returning to an area where we ats in 


agreement with our critics. Collins ends his paper by observing that, 


given the substantial progress in science studies over the last ten to 
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fifteen years, itis time for the results to be “cashed in’ for their policy 
implications. We agree entirely, although itis important that 


policymakers are not ‘short changed’ in the process: We would, 


however, offer two further notes of caution:. First, as with any 


‘applied’ research, prior ‘market research’ is absolutely essential — 
in this case, to establish what are the needs of science policymakers. 
It cannot be assumed that what is most interesting from a sociological 


point of view is necessarily most relevant from a policy perspective: 


Secondly, sociologists and others in the science studies community 


will have to do rather more than weave fairy tales if they are to 


- - Finally,-on a more personal note, we warmly welcome the fact that 
Harry Collins, in discussing what constitutes ‘the best way for science 
policy to make use of recent work in science studies’*’ seems at long 
last_to_be renouncing, implicitly at least. _ultra-relativism. We 


applaud this courageous step ‘forward’. since science policy could 
potentially derive great benefit from the qualitative sociology of 
science if the two sides were to work more closely together than 
hitherto. : 
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